Cassandra Frequent Read Write Timeouts -


i had changed whole codebase thrift cql using datastax java driver 1.0.1 , cassandra 1.2.6..

with thrift getting frequent timeouts start, not able proceed...adopting cql, tables designed per got success , less timeouts....

with able insert huge data not working thrift...but after stage, data folder around 3.5gb. getting frequent write timeout exceptions. same earlier working use case again throws timeout exception now. random once worked not working again after fresh setup.

cassadnra server log

this cassandra server partial log debug mode @ time got error :

http://pastebin.com/rw0b4md0

client exception :

caused by: com.datastax.driver.core.exceptions.writetimeoutexception: cassandra timeout during write query @ consistency 1 (1 replica required 0 acknowledged write)     @ com.datastax.driver.core.exceptions.writetimeoutexception.copy(writetimeoutexception.java:54)     @ com.datastax.driver.core.resultsetfuture.extractcausefromexecutionexception(resultsetfuture.java:214)     @ com.datastax.driver.core.resultsetfuture.getuninterruptibly(resultsetfuture.java:169)     @ com.datastax.driver.core.session.execute(session.java:107)     @ com.datastax.driver.core.session.execute(session.java:76)

infrastructure : 16gb machine 8gb heap given cassandra, i7 processor.. using single node cassandra yaml tweaked timeout, else default :

  • read_request_timeout_in_ms: 30000
  • range_request_timeout_in_ms: 30000
  • write_request_timeout_in_ms: 30000
  • truncate_request_timeout_in_ms: 60000
  • request_timeout_in_ms: 30000

use case : running usecase stores combinations(my project terminology) in cassandra....currently testing storing 250 000 combinations 100 parallel threads..each thread storing 1 combination...real case need support of tens of millions need different hardware , multi node cluster...

in storing 1 combination takes around 2sec , involves:

  • 527 insert queries
  • 506 update queries
  • 954 select queries

100 parallel threads parallel storing 100 combinations.

i had found behaviour of write timeouts random time works till 200 000 throw timeouts , not work 10k combinations. random behaviour.

i found during cassandra-stress read operations, if set rate threads high cl error. consider lower during test number of threads affordable pool sustain in order beat

  • read_request_timeout_in_ms

in opinion modifying in cassandra.yaml not idea. consider hardware resources machines work with.

for egg :

cassandra-stress read n=100000 cl=one -rate threads=200 -node n1

will give me error, while

cassandra-stress read n=100000 cl=one -rate threads=121 -node n1

will smoothly job.

hope can guys.

p.s. when read tests try spread reads on data '-pop dist=uniform(1..1000000)' or how want.


Comments

Post a Comment

Popular posts from this blog

image - ClassNotFoundException when add a prebuilt apk into system.img in android -

i'm in trouble add prebuilt apk system.img based taiwan mtk android phone platform, when push apk onto device,after reboot phone,the apk can installed succecessfully,but if add apk alps/out/target/product/$prj/system/app or vendor/mediatek/$prj/artifacts/out/target/product/$prj/system/app ,and produce system.img,the apk can't installed normally,the logcat say: a.lang.runtimeexception: unable instantiate activity componentinfo{com.sec.android.widgetapp.ap.hero.accuweather.widget/com.sec.android.widgetapp.ap.hero.accuweather.widget.weatherclock}: java.lang.classnotfoundexception: com.sec.android.widgetapp.ap.hero.accuweather.widget.weatherclock. can tell me what's problem ? when package manager installs apps part of system partition different installing apps on data partition. looks app using libraries somewhere else. when install on data lib directory created in apps data directory , symlinked /data/app-lib/. when apps installed on system it's expecte...
Read more

I need to import mysql 5.1 to 5.5? -

i have mysql 5.1 database , need import localhost running 5.5. how import 5.5 or downgrade mysql? install phpmyadmin 5.1 os: http://www.wikihow.com/install-phpmyadmin-on-your-windows-pc or https://www.digitalocean.com/community/articles/how-to-install-and-secure-phpmyadmin-on-ubuntu-12-04 then login, click on database, hit export, , export in sql format (quick fine). import .sql file 5.5 server (after installing phpmyadmin) logging in clicking database , hit import. hope helps!
Read more

Java, Hibernate, MySQL - store UTC date-time -

i saw here on few related questions (like this one , of course, this one )... essentially, want store date-time utc, , let application user choose time zone wants display date-time in. since seems date-time fields affected underlying jdbc driver, wonder if acceptable way go storing utc date-time: set both mysql , application server machine utc time zone (no need separate) both mysql , jvm should pick underlying system time settings (if not instructed otherwise) use datetime table columns on mysql side use java.util.date corresponding mapping on hibernate side (i guess java.sql.timestamp used too) let application worry interpreting date-time fields - i.e. let user choose preferred time zone is ok? edit clarify - here meant refer timestamps created strictly on server (e.g.date-time of record creation). application server instantiates date objects (new date() equals current date-time on server, , time zone agnostic). if client user wants supply date searching/filte...
Read more