c++ - Reading multiple delimited protobuf messages from a file on Windows -


i'm writing tool master thesis, needs read protobuf datastreams file. until worked exclusively on mac os , fine, i'm trying run tool on windows too.

sadly on windows not able read multiple consecutive messages single stream. tried narrow problem down , came following small program reproduces problem.

#include "tokens.pb.h" #include <google/protobuf/io/coded_stream.h> #include <google/protobuf/io/zero_copy_stream_impl.h> #include <fstream>  int main(int argc, char* argv[]) {   std::fstream tokenfile(argv[1], std::ios_base::in);   if(!tokenfile.is_open())     return -1;   google::protobuf::io::istreaminputstream iis(&tokenfile);   google::protobuf::io::codedinputstream cis(&iis);    while(true){     google::protobuf::io::codedinputstream::limit l;     unsigned int msgsize;     if(!cis.readvarint32(&msgsize))       return 0; // reached eof     l = cis.pushlimit(msgsize);      tokenio::union msg;     if(!msg.parsefromcodedstream(&cis))       return -2; // couldn't read msg      if(cis.bytesuntillimit() > 0)       return -3; // msg not read      cis.poplimit(l);      if(!msg.has_string() &&        !msg.has_file() &&        !msg.has_token() &&        !msg.has_type())       return -4; // msg contains no data   }   return 0; }

on mac os runs fine , returns 0 after reading whole file expected.

on windows first message read without problems. second messageparsefromcodedinputstream still returns true not read data. results in bytesuntillimit value larger 0 , return value of -3. of course message not contain useable data. further reads cis fail, if end of stream reached, though file has not been read completely.

i tried using fileinputstream file descriptor input same result. removing push/poplimit , reading data using readstring calls explicit message sizes , parsing string didn't help.

the following protobuf file used.

package tokenio;  message tokentype {     required uint32 id   = 1;     required string name = 2; }  message stringinstance {     required string value = 1;     optional uint64 id    = 2; }  message beginoffile {     required uint64 name = 1;     optional uint64 type = 2; }  message token {     required uint32 type   = 1;     required uint32 offset = 2;     optional uint32 line   = 3;     optional uint32 column = 4;     optional uint64 value  = 5; }  message union {     optional tokentype      type   = 1;     optional stringinstance string = 2;     optional beginoffile    file   = 3;     optional token          token  = 4; }

and this sample input file.

the input file seems ok. @ least readable protobuf editor (on windows , mac os) c++ implementation on mac os.

the code tested:

  • as working on mac os 10.8.4, compiled xcode 4.6.3 , protobuf 2.5.0
  • as not working on windows 8 64bit, compiled visual studio 2012 ultimate , protobuf 2.5.0

what doing wrong?

make std::fstream tokenfile(argv[1], std::ios_base::in | std::ios_base::binary);. default text mode; on mac , other unix-like systems doesn't matter, on windows in text mode crlf sequences translated lf, , ^z (aka '\x1a') character treated end-of-file indicator. characters might, coincidence, occur in binary stream, , cause trouble.


Comments

Post a Comment

Popular posts from this blog

image - ClassNotFoundException when add a prebuilt apk into system.img in android -

i'm in trouble add prebuilt apk system.img based taiwan mtk android phone platform, when push apk onto device,after reboot phone,the apk can installed succecessfully,but if add apk alps/out/target/product/$prj/system/app or vendor/mediatek/$prj/artifacts/out/target/product/$prj/system/app ,and produce system.img,the apk can't installed normally,the logcat say: a.lang.runtimeexception: unable instantiate activity componentinfo{com.sec.android.widgetapp.ap.hero.accuweather.widget/com.sec.android.widgetapp.ap.hero.accuweather.widget.weatherclock}: java.lang.classnotfoundexception: com.sec.android.widgetapp.ap.hero.accuweather.widget.weatherclock. can tell me what's problem ? when package manager installs apps part of system partition different installing apps on data partition. looks app using libraries somewhere else. when install on data lib directory created in apps data directory , symlinked /data/app-lib/. when apps installed on system it's expecte...
Read more

I need to import mysql 5.1 to 5.5? -

i have mysql 5.1 database , need import localhost running 5.5. how import 5.5 or downgrade mysql? install phpmyadmin 5.1 os: http://www.wikihow.com/install-phpmyadmin-on-your-windows-pc or https://www.digitalocean.com/community/articles/how-to-install-and-secure-phpmyadmin-on-ubuntu-12-04 then login, click on database, hit export, , export in sql format (quick fine). import .sql file 5.5 server (after installing phpmyadmin) logging in clicking database , hit import. hope helps!
Read more

Java, Hibernate, MySQL - store UTC date-time -

i saw here on few related questions (like this one , of course, this one )... essentially, want store date-time utc, , let application user choose time zone wants display date-time in. since seems date-time fields affected underlying jdbc driver, wonder if acceptable way go storing utc date-time: set both mysql , application server machine utc time zone (no need separate) both mysql , jvm should pick underlying system time settings (if not instructed otherwise) use datetime table columns on mysql side use java.util.date corresponding mapping on hibernate side (i guess java.sql.timestamp used too) let application worry interpreting date-time fields - i.e. let user choose preferred time zone is ok? edit clarify - here meant refer timestamps created strictly on server (e.g.date-time of record creation). application server instantiates date objects (new date() equals current date-time on server, , time zone agnostic). if client user wants supply date searching/filte...
Read more