Orbiter: EU/US/AP/EDGE | Code | Config | Playbook | Escalation Procedure
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [..X.....................................X...................] duration:
3.771636598s trace:
https://jaeger.dev.storj.io/trace/2f0b656030b9f519
FAIL (2026-05-26 04:37:19 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.771660182s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)antiblip:
3 out of 60 is failed (threshold=3) [.....X..........X................X..........................] duration:
4.099914182s trace:
https://jaeger.dev.storj.io/trace/20840979a8f7088f
FAIL (2026-05-26 04:36:29 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
25083]: Uploading piece...
May 26 04:17:28.318973560 (26h13m14.10231366s) [noise=true , address=212.102.52.217:4420 , piece_id=PQM3W3VAO5T2BBPQEREGSHGZ3EYBGJ7LFORGCKUY6NTUP2NNE5PA, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.319026442 (26h13m14.102366541s) [noise=true , address=212.102.52.216:4397 , piece_id=5UNX4TPU63Q4AG726PSLVBFWX7ZZJL766ULEHEIIU3S4LG3EUGDA, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.319245079 (26h13m14.102585178s) [noise=true , address=212.102.52.213:4343 , piece_id=AUT3LEUM7MPZIL33QGT47V7QOOZTHR3A2W2S6EMKJBY4B3AUA4ZA, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.319586399 (26h13m14.102926499s) [noise=true , address=212.102.52.216:4402 , piece_id=VNTORDVWCQPSO5C2QUQUI76CBCTFB7TIQSEL7MWCCYRJ22T4WIBA, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.319879910 (26h13m14.103220009s) [noise=true , address=212.102.52.217:4436 , piece_id=SWQCVL47FBVYZFZIQVYNAD4NYDQLBBJL3S2EYVNHGS2Y54RRAN6A, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.320133124 (26h13m14.103473223s) [noise=true , address=212.102.52.215:4379 , piece_id=FBM4P5FPAP6OCVSHW5QYJOJWRXCO63HY2I6K3EYZKL6BRIUMME5Q, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.320357862 (26h13m14.103697961s) [noise=true , address=212.102.52.214:4362 , piece_id=LB7JGSWGNGU7BQX4FZQLVA2IUDQJT7ZP5PFXOQV4NLYXH7JAFRHA, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.320571960 (26h13m14.103912059s) [noise=true , address=212.102.52.216:4408 , piece_id=23VTNWQ4FKNZ4O552CZQERQ5KMMHSCP4Y46PNSMWU4HGHSDFLIOA, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.320588152 (26h13m14.103928251s) [noise=true , address=212.102.52.213:4344 , piece_id=UEGMZ7I63LXBAKRWSFQDHBZBXPCHG2NMZSGW4PFVHFQORZLRMQYA, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.321070803 (26h13m14.104410912s) [noise=true , address=212.102.52.217:4433 , piece_id=FGN6M3NBSA3ZOLHYPT3THZV4THHNFYSI2R6BDB4AWSQZRCZ2M7TQ, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.321436691 (26h13m14.10477679s) [noise=true , address=212.102.52.215:4380 , piece_id=6XJ45LCPGAIOCFHI35GODZRUNRS2Z63RF6WIKFUXV5R3GMSH7YJQ, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.321852404 (26h13m14.105192503s) [noise=true , address=212.102.52.213:4333 , piece_id=O4MDU2BWFAOHKWNQ3KS24KNT6VYDQ4MGLBGCWW4WZZTPMVBT4OCQ, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.322253419 (26h13m14.105593518s) [noise=true , address=212.102.52.214:4347 , piece_id=VPYOK5SVHEPZ36OOB7PYDAAINGIOUACK4FVDZBMJLWKMWC2D4NXQ, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.322652791 (26h13m14.1059929s) [noise=true , address=212.102.52.213:4331 , piece_id=LTU5YKIGVWFRQXGCECWGW656T6JMUDM7RRK6QD5CRGBB5IKMZXTQ, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.324215282 (26h13m14.107555381s) [noise=true , address=212.102.52.214:4349 , piece_id=HH47IVRCTHTDW4RHMJTEXGNB4ZK62NU6LDP5AS4FVXWPFINGHFNQ, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:29.390980339 (26h13m15.174320438s) [upload=25083]: Next returned nil segment
May 26 04:17:29.390993564 (26h13m15.174333663s) [upload=25083]: Waiting for error group managing segments...
May 26 04:17:29.673854379 (26h13m15.457194479s) [noise=true , address=212.102.52.215:4381 , piece_id=5UZ45SYYXIOMUYBHLRRP6NECZCQ5JW2RIK5MQIV2LMYQBVOON4ZA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.674402245 (26h13m15.457742344s) [noise=true , address=212.102.52.217:4433 , piece_id=FGN6M3NBSA3ZOLHYPT3THZV4THHNFYSI2R6BDB4AWSQZRCZ2M7TQ, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.677827470 (26h13m15.461167569s) [noise=true , address=212.102.52.213:4344 , piece_id=UEGMZ7I63LXBAKRWSFQDHBZBXPCHG2NMZSGW4PFVHFQORZLRMQYA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.678368041 (26h13m15.46170814s) [noise=true , address=212.102.52.215:4379 , piece_id=FBM4P5FPAP6OCVSHW5QYJOJWRXCO63HY2I6K3EYZKL6BRIUMME5Q, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.680215035 (26h13m15.463555134s) [noise=true , address=212.102.52.216:4402 , piece_id=VNTORDVWCQPSO5C2QUQUI76CBCTFB7TIQSEL7MWCCYRJ22T4WIBA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.682230351 (26h13m15.46557045s) [noise=true , address=212.102.52.217:4420 , piece_id=PQM3W3VAO5T2BBPQEREGSHGZ3EYBGJ7LFORGCKUY6NTUP2NNE5PA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.685255182 (26h13m15.468595281s) [noise=true , address=212.102.52.216:4405 , piece_id=65IGD4H7EMWEDYW2TFH643JJ5QW2BKHVFZA6D77MCHRYA6T6ESGQ, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.686472445 (26h13m15.469812544s) [noise=true , address=212.102.52.214:4362 , piece_id=LB7JGSWGNGU7BQX4FZQLVA2IUDQJT7ZP5PFXOQV4NLYXH7JAFRHA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.689410080 (26h13m15.472750179s) [noise=true , address=212.102.52.217:4436 , piece_id=SWQCVL47FBVYZFZIQVYNAD4NYDQLBBJL3S2EYVNHGS2Y54RRAN6A, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.689953157 (26h13m15.473293256s) [noise=true , address=212.102.52.214:4355 , piece_id=RKSUNVZS66X4A7ZY7WEUQXSWOTK2H5VAR7TBDICVC7MGVHA7LTGA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.694371046 (26h13m15.477711155s) [noise=true , address=212.102.52.214:4347 , piece_id=VPYOK5SVHEPZ36OOB7PYDAAINGIOUACK4FVDZBMJLWKMWC2D4NXQ, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.698718109 (26h13m15.482058208s) [noise=true , address=212.102.52.214:4349 , piece_id=HH47IVRCTHTDW4RHMJTEXGNB4ZK62NU6LDP5AS4FVXWPFINGHFNQ, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.699290191 (26h13m15.48263029s) [noise=true , address=212.102.52.215:4389 , piece_id=2OTWYJIV6A2646HAW5TWBTSQVT25V2PBKVANLGGK5SWNVHE3WGWA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.701256312 (26h13m15.484596411s) [noise=true , address=212.102.52.216:4397 , piece_id=5UNX4TPU63Q4AG726PSLVBFWX7ZZJL766ULEHEIIU3S4LG3EUGDA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.735255313 (26h13m15.518595412s) [noise=true , address=212.102.52.213:4331 , piece_id=LTU5YKIGVWFRQXGCECWGW656T6JMUDM7RRK6QD5CRGBB5IKMZXTQ, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.738998335 (26h13m15.522338434s) [noise=true , address=212.102.52.215:4380 , piece_id=6XJ45LCPGAIOCFHI35GODZRUNRS2Z63RF6WIKFUXV5R3GMSH7YJQ, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.739008105 (26h13m15.522348204s) [seg_pos={0 0}, upload=25083]: Satisfied cohort requirements after 16 pieces
May 26 04:17:29.739201553 (26h13m15.522541652s) [noise=true , address=212.102.52.217:4423 , piece_id=2KWQV3P3UYSFMBJ5BJNIHGVNZZXSCD3DEJXUV7SYVDALJSXIJAPA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TgGtX5XRdWH1qn6V5fZpQ8cY5oA2NyZx1AM17XANmr99AG9oh): protocol: expected piece hash; context canceled; EOF
May 26 04:17:29.739208446 (26h13m15.522548545s) [noise=true , address=212.102.52.217:4423 , piece_id=2KWQV3P3UYSFMBJ5BJNIHGVNZZXSCD3DEJXUV7SYVDALJSXIJAPA, seg_pos={0 0}, upload=25083]: Long tail cancellation due to optimal threshold
May 26 04:17:29.739277749 (26h13m15.522617858s) [noise=true , address=212.102.52.213:4343 , piece_id=AUT3LEUM7MPZIL33QGT47V7QOOZTHR3A2W2S6EMKJBY4B3AUA4ZA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129SbAN2aTSXzo8JLELV2ssdkmT5k7kbTcvxSZE4sFsVaKw1Aok): protocol: expected piece hash; context canceled; EOF
May 26 04:17:29.739288459 (26h13m15.522628598s) [noise=true , address=212.102.52.213:4343 , piece_id=AUT3LEUM7MPZIL33QGT47V7QOOZTHR3A2W2S6EMKJBY4B3AUA4ZA, seg_pos={0 0}, upload=25083]: Long tail cancellation due to optimal threshold
May 26 04:17:29.739324538 (26h13m15.522664637s) [noise=true , address=212.102.52.216:4408 , piece_id=23VTNWQ4FKNZ4O552CZQERQ5KMMHSCP4Y46PNSMWU4HGHSDFLIOA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BsdeZy1odVCAgcjxU6KdveXhUUTJKviRrrHLNCUuVergW6zVk): protocol: expected piece hash; context canceled; EOF
May 26 04:17:29.739330078 (26h13m15.522670177s) [noise=true , address=212.102.52.216:4408 , piece_id=23VTNWQ4FKNZ4O552CZQERQ5KMMHSCP4Y46PNSMWU4HGHSDFLIOA, seg_pos={0 0}, upload=25083]: Long tail cancellation due to optimal threshold
May 26 04:17:29.739420791 (26h13m15.52276089s) [noise=true , address=212.102.52.213:4333 , piece_id=O4MDU2BWFAOHKWNQ3KS24KNT6VYDQ4MGLBGCWW4WZZTPMVBT4OCQ, seg_pos={0 0}, upload=25083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Gm3eZxZPxPeHqLQ32kpXMpCgnwRoP8epBYtiFF1EH3U9LEazY): protocol: expected piece hash; context canceled; EOF
May 26 04:17:29.739426021 (26h13m15.52276612s) [noise=true , address=212.102.52.213:4333 , piece_id=O4MDU2BWFAOHKWNQ3KS24KNT6VYDQ4MGLBGCWW4WZZTPMVBT4OCQ, seg_pos={0 0}, upload=25083]: Long tail cancellation due to optimal threshold
May 26 04:17:29.739475536 (26h13m15.522815635s) [seg_pos={0 0}, upload=25083]: Done waiting for segment. successful: 16 optimal: 16 errs: <nil>
May 26 04:17:29.739535340 (26h13m15.522875439s) [upload=25083]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 26 04:17:29.739552654 (26h13m15.522892753s) [upload=25083]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 26 04:17:39.024892360 (26h13m24.808232459s) [upload=25083]: Done uploading segments...
May 26 04:17:39.024959289 (26h13m24.808299388s) [upload=25083]: Upload finished. err: metaclient: context deadline exceeded
Duration: 4.099940733s
core
uplink
uk-select-1
uk1-placement-17
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [...X.....................................X..................] duration:
3.741684161s trace:
https://jaeger.dev.storj.io/trace/2f0b656030b9f519
FAIL (2026-05-26 04:36:10 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.741708929s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)antiblip:
3 out of 60 is failed (threshold=3) [......X..........X................X.........................] duration:
3.997848935s trace:
https://jaeger.dev.storj.io/trace/20840979a8f7088f
FAIL (2026-05-26 04:35:41 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
25083]: Uploading piece...
May 26 04:17:28.318973560 (26h13m14.10231366s) [noise=true , address=212.102.52.217:4420 , piece_id=PQM3W3VAO5T2BBPQEREGSHGZ3EYBGJ7LFORGCKUY6NTUP2NNE5PA, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.319026442 (26h13m14.102366541s) [noise=true , address=212.102.52.216:4397 , piece_id=5UNX4TPU63Q4AG726PSLVBFWX7ZZJL766ULEHEIIU3S4LG3EUGDA, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.319245079 (26h13m14.102585178s) [noise=true , address=212.102.52.213:4343 , piece_id=AUT3LEUM7MPZIL33QGT47V7QOOZTHR3A2W2S6EMKJBY4B3AUA4ZA, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.319586399 (26h13m14.102926499s) [noise=true , address=212.102.52.216:4402 , piece_id=VNTORDVWCQPSO5C2QUQUI76CBCTFB7TIQSEL7MWCCYRJ22T4WIBA, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.319879910 (26h13m14.103220009s) [noise=true , address=212.102.52.217:4436 , piece_id=SWQCVL47FBVYZFZIQVYNAD4NYDQLBBJL3S2EYVNHGS2Y54RRAN6A, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.320133124 (26h13m14.103473223s) [noise=true , address=212.102.52.215:4379 , piece_id=FBM4P5FPAP6OCVSHW5QYJOJWRXCO63HY2I6K3EYZKL6BRIUMME5Q, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.320357862 (26h13m14.103697961s) [noise=true , address=212.102.52.214:4362 , piece_id=LB7JGSWGNGU7BQX4FZQLVA2IUDQJT7ZP5PFXOQV4NLYXH7JAFRHA, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.320571960 (26h13m14.103912059s) [noise=true , address=212.102.52.216:4408 , piece_id=23VTNWQ4FKNZ4O552CZQERQ5KMMHSCP4Y46PNSMWU4HGHSDFLIOA, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.320588152 (26h13m14.103928251s) [noise=true , address=212.102.52.213:4344 , piece_id=UEGMZ7I63LXBAKRWSFQDHBZBXPCHG2NMZSGW4PFVHFQORZLRMQYA, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.321070803 (26h13m14.104410912s) [noise=true , address=212.102.52.217:4433 , piece_id=FGN6M3NBSA3ZOLHYPT3THZV4THHNFYSI2R6BDB4AWSQZRCZ2M7TQ, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.321436691 (26h13m14.10477679s) [noise=true , address=212.102.52.215:4380 , piece_id=6XJ45LCPGAIOCFHI35GODZRUNRS2Z63RF6WIKFUXV5R3GMSH7YJQ, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.321852404 (26h13m14.105192503s) [noise=true , address=212.102.52.213:4333 , piece_id=O4MDU2BWFAOHKWNQ3KS24KNT6VYDQ4MGLBGCWW4WZZTPMVBT4OCQ, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.322253419 (26h13m14.105593518s) [noise=true , address=212.102.52.214:4347 , piece_id=VPYOK5SVHEPZ36OOB7PYDAAINGIOUACK4FVDZBMJLWKMWC2D4NXQ, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.322652791 (26h13m14.1059929s) [noise=true , address=212.102.52.213:4331 , piece_id=LTU5YKIGVWFRQXGCECWGW656T6JMUDM7RRK6QD5CRGBB5IKMZXTQ, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.324215282 (26h13m14.107555381s) [noise=true , address=212.102.52.214:4349 , piece_id=HH47IVRCTHTDW4RHMJTEXGNB4ZK62NU6LDP5AS4FVXWPFINGHFNQ, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:29.390980339 (26h13m15.174320438s) [upload=25083]: Next returned nil segment
May 26 04:17:29.390993564 (26h13m15.174333663s) [upload=25083]: Waiting for error group managing segments...
May 26 04:17:29.673854379 (26h13m15.457194479s) [noise=true , address=212.102.52.215:4381 , piece_id=5UZ45SYYXIOMUYBHLRRP6NECZCQ5JW2RIK5MQIV2LMYQBVOON4ZA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.674402245 (26h13m15.457742344s) [noise=true , address=212.102.52.217:4433 , piece_id=FGN6M3NBSA3ZOLHYPT3THZV4THHNFYSI2R6BDB4AWSQZRCZ2M7TQ, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.677827470 (26h13m15.461167569s) [noise=true , address=212.102.52.213:4344 , piece_id=UEGMZ7I63LXBAKRWSFQDHBZBXPCHG2NMZSGW4PFVHFQORZLRMQYA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.678368041 (26h13m15.46170814s) [noise=true , address=212.102.52.215:4379 , piece_id=FBM4P5FPAP6OCVSHW5QYJOJWRXCO63HY2I6K3EYZKL6BRIUMME5Q, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.680215035 (26h13m15.463555134s) [noise=true , address=212.102.52.216:4402 , piece_id=VNTORDVWCQPSO5C2QUQUI76CBCTFB7TIQSEL7MWCCYRJ22T4WIBA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.682230351 (26h13m15.46557045s) [noise=true , address=212.102.52.217:4420 , piece_id=PQM3W3VAO5T2BBPQEREGSHGZ3EYBGJ7LFORGCKUY6NTUP2NNE5PA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.685255182 (26h13m15.468595281s) [noise=true , address=212.102.52.216:4405 , piece_id=65IGD4H7EMWEDYW2TFH643JJ5QW2BKHVFZA6D77MCHRYA6T6ESGQ, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.686472445 (26h13m15.469812544s) [noise=true , address=212.102.52.214:4362 , piece_id=LB7JGSWGNGU7BQX4FZQLVA2IUDQJT7ZP5PFXOQV4NLYXH7JAFRHA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.689410080 (26h13m15.472750179s) [noise=true , address=212.102.52.217:4436 , piece_id=SWQCVL47FBVYZFZIQVYNAD4NYDQLBBJL3S2EYVNHGS2Y54RRAN6A, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.689953157 (26h13m15.473293256s) [noise=true , address=212.102.52.214:4355 , piece_id=RKSUNVZS66X4A7ZY7WEUQXSWOTK2H5VAR7TBDICVC7MGVHA7LTGA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.694371046 (26h13m15.477711155s) [noise=true , address=212.102.52.214:4347 , piece_id=VPYOK5SVHEPZ36OOB7PYDAAINGIOUACK4FVDZBMJLWKMWC2D4NXQ, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.698718109 (26h13m15.482058208s) [noise=true , address=212.102.52.214:4349 , piece_id=HH47IVRCTHTDW4RHMJTEXGNB4ZK62NU6LDP5AS4FVXWPFINGHFNQ, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.699290191 (26h13m15.48263029s) [noise=true , address=212.102.52.215:4389 , piece_id=2OTWYJIV6A2646HAW5TWBTSQVT25V2PBKVANLGGK5SWNVHE3WGWA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.701256312 (26h13m15.484596411s) [noise=true , address=212.102.52.216:4397 , piece_id=5UNX4TPU63Q4AG726PSLVBFWX7ZZJL766ULEHEIIU3S4LG3EUGDA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.735255313 (26h13m15.518595412s) [noise=true , address=212.102.52.213:4331 , piece_id=LTU5YKIGVWFRQXGCECWGW656T6JMUDM7RRK6QD5CRGBB5IKMZXTQ, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.738998335 (26h13m15.522338434s) [noise=true , address=212.102.52.215:4380 , piece_id=6XJ45LCPGAIOCFHI35GODZRUNRS2Z63RF6WIKFUXV5R3GMSH7YJQ, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.739008105 (26h13m15.522348204s) [seg_pos={0 0}, upload=25083]: Satisfied cohort requirements after 16 pieces
May 26 04:17:29.739201553 (26h13m15.522541652s) [noise=true , address=212.102.52.217:4423 , piece_id=2KWQV3P3UYSFMBJ5BJNIHGVNZZXSCD3DEJXUV7SYVDALJSXIJAPA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TgGtX5XRdWH1qn6V5fZpQ8cY5oA2NyZx1AM17XANmr99AG9oh): protocol: expected piece hash; context canceled; EOF
May 26 04:17:29.739208446 (26h13m15.522548545s) [noise=true , address=212.102.52.217:4423 , piece_id=2KWQV3P3UYSFMBJ5BJNIHGVNZZXSCD3DEJXUV7SYVDALJSXIJAPA, seg_pos={0 0}, upload=25083]: Long tail cancellation due to optimal threshold
May 26 04:17:29.739277749 (26h13m15.522617858s) [noise=true , address=212.102.52.213:4343 , piece_id=AUT3LEUM7MPZIL33QGT47V7QOOZTHR3A2W2S6EMKJBY4B3AUA4ZA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129SbAN2aTSXzo8JLELV2ssdkmT5k7kbTcvxSZE4sFsVaKw1Aok): protocol: expected piece hash; context canceled; EOF
May 26 04:17:29.739288459 (26h13m15.522628598s) [noise=true , address=212.102.52.213:4343 , piece_id=AUT3LEUM7MPZIL33QGT47V7QOOZTHR3A2W2S6EMKJBY4B3AUA4ZA, seg_pos={0 0}, upload=25083]: Long tail cancellation due to optimal threshold
May 26 04:17:29.739324538 (26h13m15.522664637s) [noise=true , address=212.102.52.216:4408 , piece_id=23VTNWQ4FKNZ4O552CZQERQ5KMMHSCP4Y46PNSMWU4HGHSDFLIOA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BsdeZy1odVCAgcjxU6KdveXhUUTJKviRrrHLNCUuVergW6zVk): protocol: expected piece hash; context canceled; EOF
May 26 04:17:29.739330078 (26h13m15.522670177s) [noise=true , address=212.102.52.216:4408 , piece_id=23VTNWQ4FKNZ4O552CZQERQ5KMMHSCP4Y46PNSMWU4HGHSDFLIOA, seg_pos={0 0}, upload=25083]: Long tail cancellation due to optimal threshold
May 26 04:17:29.739420791 (26h13m15.52276089s) [noise=true , address=212.102.52.213:4333 , piece_id=O4MDU2BWFAOHKWNQ3KS24KNT6VYDQ4MGLBGCWW4WZZTPMVBT4OCQ, seg_pos={0 0}, upload=25083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Gm3eZxZPxPeHqLQ32kpXMpCgnwRoP8epBYtiFF1EH3U9LEazY): protocol: expected piece hash; context canceled; EOF
May 26 04:17:29.739426021 (26h13m15.52276612s) [noise=true , address=212.102.52.213:4333 , piece_id=O4MDU2BWFAOHKWNQ3KS24KNT6VYDQ4MGLBGCWW4WZZTPMVBT4OCQ, seg_pos={0 0}, upload=25083]: Long tail cancellation due to optimal threshold
May 26 04:17:29.739475536 (26h13m15.522815635s) [seg_pos={0 0}, upload=25083]: Done waiting for segment. successful: 16 optimal: 16 errs: <nil>
May 26 04:17:29.739535340 (26h13m15.522875439s) [upload=25083]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 26 04:17:29.739552654 (26h13m15.522892753s) [upload=25083]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 26 04:17:39.024892360 (26h13m24.808232459s) [upload=25083]: Done uploading segments...
May 26 04:17:39.024959289 (26h13m24.808299388s) [upload=25083]: Upload finished. err: metaclient: context deadline exceeded
Duration: 3.997863113s
core
uplink
uk-select-1
uk1-placement-17
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [....X.....................................X.................] duration:
4.712274294s trace:
https://jaeger.dev.storj.io/trace/2f0b656030b9f519
FAIL (2026-05-26 04:35:40 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 4.71229787s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)antiblip:
3 out of 60 is failed (threshold=3) [.......X..........X................X........................] duration:
51.168885098s trace:
https://jaeger.dev.storj.io/trace/20840979a8f7088f
FAIL (2026-05-26 04:35:37 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
25083]: Uploading piece...
May 26 04:17:28.318973560 (26h13m14.10231366s) [noise=true , address=212.102.52.217:4420 , piece_id=PQM3W3VAO5T2BBPQEREGSHGZ3EYBGJ7LFORGCKUY6NTUP2NNE5PA, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.319026442 (26h13m14.102366541s) [noise=true , address=212.102.52.216:4397 , piece_id=5UNX4TPU63Q4AG726PSLVBFWX7ZZJL766ULEHEIIU3S4LG3EUGDA, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.319245079 (26h13m14.102585178s) [noise=true , address=212.102.52.213:4343 , piece_id=AUT3LEUM7MPZIL33QGT47V7QOOZTHR3A2W2S6EMKJBY4B3AUA4ZA, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.319586399 (26h13m14.102926499s) [noise=true , address=212.102.52.216:4402 , piece_id=VNTORDVWCQPSO5C2QUQUI76CBCTFB7TIQSEL7MWCCYRJ22T4WIBA, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.319879910 (26h13m14.103220009s) [noise=true , address=212.102.52.217:4436 , piece_id=SWQCVL47FBVYZFZIQVYNAD4NYDQLBBJL3S2EYVNHGS2Y54RRAN6A, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.320133124 (26h13m14.103473223s) [noise=true , address=212.102.52.215:4379 , piece_id=FBM4P5FPAP6OCVSHW5QYJOJWRXCO63HY2I6K3EYZKL6BRIUMME5Q, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.320357862 (26h13m14.103697961s) [noise=true , address=212.102.52.214:4362 , piece_id=LB7JGSWGNGU7BQX4FZQLVA2IUDQJT7ZP5PFXOQV4NLYXH7JAFRHA, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.320571960 (26h13m14.103912059s) [noise=true , address=212.102.52.216:4408 , piece_id=23VTNWQ4FKNZ4O552CZQERQ5KMMHSCP4Y46PNSMWU4HGHSDFLIOA, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.320588152 (26h13m14.103928251s) [noise=true , address=212.102.52.213:4344 , piece_id=UEGMZ7I63LXBAKRWSFQDHBZBXPCHG2NMZSGW4PFVHFQORZLRMQYA, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.321070803 (26h13m14.104410912s) [noise=true , address=212.102.52.217:4433 , piece_id=FGN6M3NBSA3ZOLHYPT3THZV4THHNFYSI2R6BDB4AWSQZRCZ2M7TQ, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.321436691 (26h13m14.10477679s) [noise=true , address=212.102.52.215:4380 , piece_id=6XJ45LCPGAIOCFHI35GODZRUNRS2Z63RF6WIKFUXV5R3GMSH7YJQ, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.321852404 (26h13m14.105192503s) [noise=true , address=212.102.52.213:4333 , piece_id=O4MDU2BWFAOHKWNQ3KS24KNT6VYDQ4MGLBGCWW4WZZTPMVBT4OCQ, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.322253419 (26h13m14.105593518s) [noise=true , address=212.102.52.214:4347 , piece_id=VPYOK5SVHEPZ36OOB7PYDAAINGIOUACK4FVDZBMJLWKMWC2D4NXQ, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.322652791 (26h13m14.1059929s) [noise=true , address=212.102.52.213:4331 , piece_id=LTU5YKIGVWFRQXGCECWGW656T6JMUDM7RRK6QD5CRGBB5IKMZXTQ, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.324215282 (26h13m14.107555381s) [noise=true , address=212.102.52.214:4349 , piece_id=HH47IVRCTHTDW4RHMJTEXGNB4ZK62NU6LDP5AS4FVXWPFINGHFNQ, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:29.390980339 (26h13m15.174320438s) [upload=25083]: Next returned nil segment
May 26 04:17:29.390993564 (26h13m15.174333663s) [upload=25083]: Waiting for error group managing segments...
May 26 04:17:29.673854379 (26h13m15.457194479s) [noise=true , address=212.102.52.215:4381 , piece_id=5UZ45SYYXIOMUYBHLRRP6NECZCQ5JW2RIK5MQIV2LMYQBVOON4ZA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.674402245 (26h13m15.457742344s) [noise=true , address=212.102.52.217:4433 , piece_id=FGN6M3NBSA3ZOLHYPT3THZV4THHNFYSI2R6BDB4AWSQZRCZ2M7TQ, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.677827470 (26h13m15.461167569s) [noise=true , address=212.102.52.213:4344 , piece_id=UEGMZ7I63LXBAKRWSFQDHBZBXPCHG2NMZSGW4PFVHFQORZLRMQYA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.678368041 (26h13m15.46170814s) [noise=true , address=212.102.52.215:4379 , piece_id=FBM4P5FPAP6OCVSHW5QYJOJWRXCO63HY2I6K3EYZKL6BRIUMME5Q, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.680215035 (26h13m15.463555134s) [noise=true , address=212.102.52.216:4402 , piece_id=VNTORDVWCQPSO5C2QUQUI76CBCTFB7TIQSEL7MWCCYRJ22T4WIBA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.682230351 (26h13m15.46557045s) [noise=true , address=212.102.52.217:4420 , piece_id=PQM3W3VAO5T2BBPQEREGSHGZ3EYBGJ7LFORGCKUY6NTUP2NNE5PA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.685255182 (26h13m15.468595281s) [noise=true , address=212.102.52.216:4405 , piece_id=65IGD4H7EMWEDYW2TFH643JJ5QW2BKHVFZA6D77MCHRYA6T6ESGQ, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.686472445 (26h13m15.469812544s) [noise=true , address=212.102.52.214:4362 , piece_id=LB7JGSWGNGU7BQX4FZQLVA2IUDQJT7ZP5PFXOQV4NLYXH7JAFRHA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.689410080 (26h13m15.472750179s) [noise=true , address=212.102.52.217:4436 , piece_id=SWQCVL47FBVYZFZIQVYNAD4NYDQLBBJL3S2EYVNHGS2Y54RRAN6A, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.689953157 (26h13m15.473293256s) [noise=true , address=212.102.52.214:4355 , piece_id=RKSUNVZS66X4A7ZY7WEUQXSWOTK2H5VAR7TBDICVC7MGVHA7LTGA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.694371046 (26h13m15.477711155s) [noise=true , address=212.102.52.214:4347 , piece_id=VPYOK5SVHEPZ36OOB7PYDAAINGIOUACK4FVDZBMJLWKMWC2D4NXQ, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.698718109 (26h13m15.482058208s) [noise=true , address=212.102.52.214:4349 , piece_id=HH47IVRCTHTDW4RHMJTEXGNB4ZK62NU6LDP5AS4FVXWPFINGHFNQ, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.699290191 (26h13m15.48263029s) [noise=true , address=212.102.52.215:4389 , piece_id=2OTWYJIV6A2646HAW5TWBTSQVT25V2PBKVANLGGK5SWNVHE3WGWA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.701256312 (26h13m15.484596411s) [noise=true , address=212.102.52.216:4397 , piece_id=5UNX4TPU63Q4AG726PSLVBFWX7ZZJL766ULEHEIIU3S4LG3EUGDA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.735255313 (26h13m15.518595412s) [noise=true , address=212.102.52.213:4331 , piece_id=LTU5YKIGVWFRQXGCECWGW656T6JMUDM7RRK6QD5CRGBB5IKMZXTQ, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.738998335 (26h13m15.522338434s) [noise=true , address=212.102.52.215:4380 , piece_id=6XJ45LCPGAIOCFHI35GODZRUNRS2Z63RF6WIKFUXV5R3GMSH7YJQ, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.739008105 (26h13m15.522348204s) [seg_pos={0 0}, upload=25083]: Satisfied cohort requirements after 16 pieces
May 26 04:17:29.739201553 (26h13m15.522541652s) [noise=true , address=212.102.52.217:4423 , piece_id=2KWQV3P3UYSFMBJ5BJNIHGVNZZXSCD3DEJXUV7SYVDALJSXIJAPA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TgGtX5XRdWH1qn6V5fZpQ8cY5oA2NyZx1AM17XANmr99AG9oh): protocol: expected piece hash; context canceled; EOF
May 26 04:17:29.739208446 (26h13m15.522548545s) [noise=true , address=212.102.52.217:4423 , piece_id=2KWQV3P3UYSFMBJ5BJNIHGVNZZXSCD3DEJXUV7SYVDALJSXIJAPA, seg_pos={0 0}, upload=25083]: Long tail cancellation due to optimal threshold
May 26 04:17:29.739277749 (26h13m15.522617858s) [noise=true , address=212.102.52.213:4343 , piece_id=AUT3LEUM7MPZIL33QGT47V7QOOZTHR3A2W2S6EMKJBY4B3AUA4ZA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129SbAN2aTSXzo8JLELV2ssdkmT5k7kbTcvxSZE4sFsVaKw1Aok): protocol: expected piece hash; context canceled; EOF
May 26 04:17:29.739288459 (26h13m15.522628598s) [noise=true , address=212.102.52.213:4343 , piece_id=AUT3LEUM7MPZIL33QGT47V7QOOZTHR3A2W2S6EMKJBY4B3AUA4ZA, seg_pos={0 0}, upload=25083]: Long tail cancellation due to optimal threshold
May 26 04:17:29.739324538 (26h13m15.522664637s) [noise=true , address=212.102.52.216:4408 , piece_id=23VTNWQ4FKNZ4O552CZQERQ5KMMHSCP4Y46PNSMWU4HGHSDFLIOA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BsdeZy1odVCAgcjxU6KdveXhUUTJKviRrrHLNCUuVergW6zVk): protocol: expected piece hash; context canceled; EOF
May 26 04:17:29.739330078 (26h13m15.522670177s) [noise=true , address=212.102.52.216:4408 , piece_id=23VTNWQ4FKNZ4O552CZQERQ5KMMHSCP4Y46PNSMWU4HGHSDFLIOA, seg_pos={0 0}, upload=25083]: Long tail cancellation due to optimal threshold
May 26 04:17:29.739420791 (26h13m15.52276089s) [noise=true , address=212.102.52.213:4333 , piece_id=O4MDU2BWFAOHKWNQ3KS24KNT6VYDQ4MGLBGCWW4WZZTPMVBT4OCQ, seg_pos={0 0}, upload=25083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Gm3eZxZPxPeHqLQ32kpXMpCgnwRoP8epBYtiFF1EH3U9LEazY): protocol: expected piece hash; context canceled; EOF
May 26 04:17:29.739426021 (26h13m15.52276612s) [noise=true , address=212.102.52.213:4333 , piece_id=O4MDU2BWFAOHKWNQ3KS24KNT6VYDQ4MGLBGCWW4WZZTPMVBT4OCQ, seg_pos={0 0}, upload=25083]: Long tail cancellation due to optimal threshold
May 26 04:17:29.739475536 (26h13m15.522815635s) [seg_pos={0 0}, upload=25083]: Done waiting for segment. successful: 16 optimal: 16 errs: <nil>
May 26 04:17:29.739535340 (26h13m15.522875439s) [upload=25083]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 26 04:17:29.739552654 (26h13m15.522892753s) [upload=25083]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 26 04:17:39.024892360 (26h13m24.808232459s) [upload=25083]: Done uploading segments...
May 26 04:17:39.024959289 (26h13m24.808299388s) [upload=25083]: Upload finished. err: metaclient: context deadline exceeded
Duration: 51.168910275s
core
uplink
uk-select-1
uk1-placement-17
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [.....X.....................................X................] duration:
3.717164386s trace:
https://jaeger.dev.storj.io/trace/2f0b656030b9f519
FAIL (2026-05-26 04:35:03 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.717183343s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [......X.....................................X...............] duration:
3.816922917s trace:
https://jaeger.dev.storj.io/trace/2f0b656030b9f519
FAIL (2026-05-26 04:34:00 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.816945882s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)antiblip:
3 out of 60 is failed (threshold=3) [........X..........X................X.......................] duration:
5.602145305s trace:
https://jaeger.dev.storj.io/trace/20840979a8f7088f
FAIL (2026-05-26 04:33:56 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
25083]: Uploading piece...
May 26 04:17:28.318973560 (26h13m14.10231366s) [noise=true , address=212.102.52.217:4420 , piece_id=PQM3W3VAO5T2BBPQEREGSHGZ3EYBGJ7LFORGCKUY6NTUP2NNE5PA, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.319026442 (26h13m14.102366541s) [noise=true , address=212.102.52.216:4397 , piece_id=5UNX4TPU63Q4AG726PSLVBFWX7ZZJL766ULEHEIIU3S4LG3EUGDA, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.319245079 (26h13m14.102585178s) [noise=true , address=212.102.52.213:4343 , piece_id=AUT3LEUM7MPZIL33QGT47V7QOOZTHR3A2W2S6EMKJBY4B3AUA4ZA, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.319586399 (26h13m14.102926499s) [noise=true , address=212.102.52.216:4402 , piece_id=VNTORDVWCQPSO5C2QUQUI76CBCTFB7TIQSEL7MWCCYRJ22T4WIBA, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.319879910 (26h13m14.103220009s) [noise=true , address=212.102.52.217:4436 , piece_id=SWQCVL47FBVYZFZIQVYNAD4NYDQLBBJL3S2EYVNHGS2Y54RRAN6A, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.320133124 (26h13m14.103473223s) [noise=true , address=212.102.52.215:4379 , piece_id=FBM4P5FPAP6OCVSHW5QYJOJWRXCO63HY2I6K3EYZKL6BRIUMME5Q, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.320357862 (26h13m14.103697961s) [noise=true , address=212.102.52.214:4362 , piece_id=LB7JGSWGNGU7BQX4FZQLVA2IUDQJT7ZP5PFXOQV4NLYXH7JAFRHA, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.320571960 (26h13m14.103912059s) [noise=true , address=212.102.52.216:4408 , piece_id=23VTNWQ4FKNZ4O552CZQERQ5KMMHSCP4Y46PNSMWU4HGHSDFLIOA, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.320588152 (26h13m14.103928251s) [noise=true , address=212.102.52.213:4344 , piece_id=UEGMZ7I63LXBAKRWSFQDHBZBXPCHG2NMZSGW4PFVHFQORZLRMQYA, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.321070803 (26h13m14.104410912s) [noise=true , address=212.102.52.217:4433 , piece_id=FGN6M3NBSA3ZOLHYPT3THZV4THHNFYSI2R6BDB4AWSQZRCZ2M7TQ, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.321436691 (26h13m14.10477679s) [noise=true , address=212.102.52.215:4380 , piece_id=6XJ45LCPGAIOCFHI35GODZRUNRS2Z63RF6WIKFUXV5R3GMSH7YJQ, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.321852404 (26h13m14.105192503s) [noise=true , address=212.102.52.213:4333 , piece_id=O4MDU2BWFAOHKWNQ3KS24KNT6VYDQ4MGLBGCWW4WZZTPMVBT4OCQ, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.322253419 (26h13m14.105593518s) [noise=true , address=212.102.52.214:4347 , piece_id=VPYOK5SVHEPZ36OOB7PYDAAINGIOUACK4FVDZBMJLWKMWC2D4NXQ, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.322652791 (26h13m14.1059929s) [noise=true , address=212.102.52.213:4331 , piece_id=LTU5YKIGVWFRQXGCECWGW656T6JMUDM7RRK6QD5CRGBB5IKMZXTQ, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.324215282 (26h13m14.107555381s) [noise=true , address=212.102.52.214:4349 , piece_id=HH47IVRCTHTDW4RHMJTEXGNB4ZK62NU6LDP5AS4FVXWPFINGHFNQ, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:29.390980339 (26h13m15.174320438s) [upload=25083]: Next returned nil segment
May 26 04:17:29.390993564 (26h13m15.174333663s) [upload=25083]: Waiting for error group managing segments...
May 26 04:17:29.673854379 (26h13m15.457194479s) [noise=true , address=212.102.52.215:4381 , piece_id=5UZ45SYYXIOMUYBHLRRP6NECZCQ5JW2RIK5MQIV2LMYQBVOON4ZA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.674402245 (26h13m15.457742344s) [noise=true , address=212.102.52.217:4433 , piece_id=FGN6M3NBSA3ZOLHYPT3THZV4THHNFYSI2R6BDB4AWSQZRCZ2M7TQ, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.677827470 (26h13m15.461167569s) [noise=true , address=212.102.52.213:4344 , piece_id=UEGMZ7I63LXBAKRWSFQDHBZBXPCHG2NMZSGW4PFVHFQORZLRMQYA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.678368041 (26h13m15.46170814s) [noise=true , address=212.102.52.215:4379 , piece_id=FBM4P5FPAP6OCVSHW5QYJOJWRXCO63HY2I6K3EYZKL6BRIUMME5Q, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.680215035 (26h13m15.463555134s) [noise=true , address=212.102.52.216:4402 , piece_id=VNTORDVWCQPSO5C2QUQUI76CBCTFB7TIQSEL7MWCCYRJ22T4WIBA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.682230351 (26h13m15.46557045s) [noise=true , address=212.102.52.217:4420 , piece_id=PQM3W3VAO5T2BBPQEREGSHGZ3EYBGJ7LFORGCKUY6NTUP2NNE5PA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.685255182 (26h13m15.468595281s) [noise=true , address=212.102.52.216:4405 , piece_id=65IGD4H7EMWEDYW2TFH643JJ5QW2BKHVFZA6D77MCHRYA6T6ESGQ, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.686472445 (26h13m15.469812544s) [noise=true , address=212.102.52.214:4362 , piece_id=LB7JGSWGNGU7BQX4FZQLVA2IUDQJT7ZP5PFXOQV4NLYXH7JAFRHA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.689410080 (26h13m15.472750179s) [noise=true , address=212.102.52.217:4436 , piece_id=SWQCVL47FBVYZFZIQVYNAD4NYDQLBBJL3S2EYVNHGS2Y54RRAN6A, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.689953157 (26h13m15.473293256s) [noise=true , address=212.102.52.214:4355 , piece_id=RKSUNVZS66X4A7ZY7WEUQXSWOTK2H5VAR7TBDICVC7MGVHA7LTGA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.694371046 (26h13m15.477711155s) [noise=true , address=212.102.52.214:4347 , piece_id=VPYOK5SVHEPZ36OOB7PYDAAINGIOUACK4FVDZBMJLWKMWC2D4NXQ, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.698718109 (26h13m15.482058208s) [noise=true , address=212.102.52.214:4349 , piece_id=HH47IVRCTHTDW4RHMJTEXGNB4ZK62NU6LDP5AS4FVXWPFINGHFNQ, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.699290191 (26h13m15.48263029s) [noise=true , address=212.102.52.215:4389 , piece_id=2OTWYJIV6A2646HAW5TWBTSQVT25V2PBKVANLGGK5SWNVHE3WGWA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.701256312 (26h13m15.484596411s) [noise=true , address=212.102.52.216:4397 , piece_id=5UNX4TPU63Q4AG726PSLVBFWX7ZZJL766ULEHEIIU3S4LG3EUGDA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.735255313 (26h13m15.518595412s) [noise=true , address=212.102.52.213:4331 , piece_id=LTU5YKIGVWFRQXGCECWGW656T6JMUDM7RRK6QD5CRGBB5IKMZXTQ, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.738998335 (26h13m15.522338434s) [noise=true , address=212.102.52.215:4380 , piece_id=6XJ45LCPGAIOCFHI35GODZRUNRS2Z63RF6WIKFUXV5R3GMSH7YJQ, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.739008105 (26h13m15.522348204s) [seg_pos={0 0}, upload=25083]: Satisfied cohort requirements after 16 pieces
May 26 04:17:29.739201553 (26h13m15.522541652s) [noise=true , address=212.102.52.217:4423 , piece_id=2KWQV3P3UYSFMBJ5BJNIHGVNZZXSCD3DEJXUV7SYVDALJSXIJAPA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TgGtX5XRdWH1qn6V5fZpQ8cY5oA2NyZx1AM17XANmr99AG9oh): protocol: expected piece hash; context canceled; EOF
May 26 04:17:29.739208446 (26h13m15.522548545s) [noise=true , address=212.102.52.217:4423 , piece_id=2KWQV3P3UYSFMBJ5BJNIHGVNZZXSCD3DEJXUV7SYVDALJSXIJAPA, seg_pos={0 0}, upload=25083]: Long tail cancellation due to optimal threshold
May 26 04:17:29.739277749 (26h13m15.522617858s) [noise=true , address=212.102.52.213:4343 , piece_id=AUT3LEUM7MPZIL33QGT47V7QOOZTHR3A2W2S6EMKJBY4B3AUA4ZA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129SbAN2aTSXzo8JLELV2ssdkmT5k7kbTcvxSZE4sFsVaKw1Aok): protocol: expected piece hash; context canceled; EOF
May 26 04:17:29.739288459 (26h13m15.522628598s) [noise=true , address=212.102.52.213:4343 , piece_id=AUT3LEUM7MPZIL33QGT47V7QOOZTHR3A2W2S6EMKJBY4B3AUA4ZA, seg_pos={0 0}, upload=25083]: Long tail cancellation due to optimal threshold
May 26 04:17:29.739324538 (26h13m15.522664637s) [noise=true , address=212.102.52.216:4408 , piece_id=23VTNWQ4FKNZ4O552CZQERQ5KMMHSCP4Y46PNSMWU4HGHSDFLIOA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BsdeZy1odVCAgcjxU6KdveXhUUTJKviRrrHLNCUuVergW6zVk): protocol: expected piece hash; context canceled; EOF
May 26 04:17:29.739330078 (26h13m15.522670177s) [noise=true , address=212.102.52.216:4408 , piece_id=23VTNWQ4FKNZ4O552CZQERQ5KMMHSCP4Y46PNSMWU4HGHSDFLIOA, seg_pos={0 0}, upload=25083]: Long tail cancellation due to optimal threshold
May 26 04:17:29.739420791 (26h13m15.52276089s) [noise=true , address=212.102.52.213:4333 , piece_id=O4MDU2BWFAOHKWNQ3KS24KNT6VYDQ4MGLBGCWW4WZZTPMVBT4OCQ, seg_pos={0 0}, upload=25083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Gm3eZxZPxPeHqLQ32kpXMpCgnwRoP8epBYtiFF1EH3U9LEazY): protocol: expected piece hash; context canceled; EOF
May 26 04:17:29.739426021 (26h13m15.52276612s) [noise=true , address=212.102.52.213:4333 , piece_id=O4MDU2BWFAOHKWNQ3KS24KNT6VYDQ4MGLBGCWW4WZZTPMVBT4OCQ, seg_pos={0 0}, upload=25083]: Long tail cancellation due to optimal threshold
May 26 04:17:29.739475536 (26h13m15.522815635s) [seg_pos={0 0}, upload=25083]: Done waiting for segment. successful: 16 optimal: 16 errs: <nil>
May 26 04:17:29.739535340 (26h13m15.522875439s) [upload=25083]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 26 04:17:29.739552654 (26h13m15.522892753s) [upload=25083]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 26 04:17:39.024892360 (26h13m24.808232459s) [upload=25083]: Done uploading segments...
May 26 04:17:39.024959289 (26h13m24.808299388s) [upload=25083]: Upload finished. err: metaclient: context deadline exceeded
Duration: 5.602162107s
core
uplink
uk-select-1
uk1-placement-17
|
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)antiblip:
3 out of 60 is failed (threshold=3) [.........X..........X................X......................] duration:
4.062531295s trace:
https://jaeger.dev.storj.io/trace/20840979a8f7088f
FAIL (2026-05-26 04:33:26 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
25083]: Uploading piece...
May 26 04:17:28.318973560 (26h13m14.10231366s) [noise=true , address=212.102.52.217:4420 , piece_id=PQM3W3VAO5T2BBPQEREGSHGZ3EYBGJ7LFORGCKUY6NTUP2NNE5PA, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.319026442 (26h13m14.102366541s) [noise=true , address=212.102.52.216:4397 , piece_id=5UNX4TPU63Q4AG726PSLVBFWX7ZZJL766ULEHEIIU3S4LG3EUGDA, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.319245079 (26h13m14.102585178s) [noise=true , address=212.102.52.213:4343 , piece_id=AUT3LEUM7MPZIL33QGT47V7QOOZTHR3A2W2S6EMKJBY4B3AUA4ZA, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.319586399 (26h13m14.102926499s) [noise=true , address=212.102.52.216:4402 , piece_id=VNTORDVWCQPSO5C2QUQUI76CBCTFB7TIQSEL7MWCCYRJ22T4WIBA, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.319879910 (26h13m14.103220009s) [noise=true , address=212.102.52.217:4436 , piece_id=SWQCVL47FBVYZFZIQVYNAD4NYDQLBBJL3S2EYVNHGS2Y54RRAN6A, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.320133124 (26h13m14.103473223s) [noise=true , address=212.102.52.215:4379 , piece_id=FBM4P5FPAP6OCVSHW5QYJOJWRXCO63HY2I6K3EYZKL6BRIUMME5Q, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.320357862 (26h13m14.103697961s) [noise=true , address=212.102.52.214:4362 , piece_id=LB7JGSWGNGU7BQX4FZQLVA2IUDQJT7ZP5PFXOQV4NLYXH7JAFRHA, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.320571960 (26h13m14.103912059s) [noise=true , address=212.102.52.216:4408 , piece_id=23VTNWQ4FKNZ4O552CZQERQ5KMMHSCP4Y46PNSMWU4HGHSDFLIOA, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.320588152 (26h13m14.103928251s) [noise=true , address=212.102.52.213:4344 , piece_id=UEGMZ7I63LXBAKRWSFQDHBZBXPCHG2NMZSGW4PFVHFQORZLRMQYA, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.321070803 (26h13m14.104410912s) [noise=true , address=212.102.52.217:4433 , piece_id=FGN6M3NBSA3ZOLHYPT3THZV4THHNFYSI2R6BDB4AWSQZRCZ2M7TQ, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.321436691 (26h13m14.10477679s) [noise=true , address=212.102.52.215:4380 , piece_id=6XJ45LCPGAIOCFHI35GODZRUNRS2Z63RF6WIKFUXV5R3GMSH7YJQ, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.321852404 (26h13m14.105192503s) [noise=true , address=212.102.52.213:4333 , piece_id=O4MDU2BWFAOHKWNQ3KS24KNT6VYDQ4MGLBGCWW4WZZTPMVBT4OCQ, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.322253419 (26h13m14.105593518s) [noise=true , address=212.102.52.214:4347 , piece_id=VPYOK5SVHEPZ36OOB7PYDAAINGIOUACK4FVDZBMJLWKMWC2D4NXQ, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.322652791 (26h13m14.1059929s) [noise=true , address=212.102.52.213:4331 , piece_id=LTU5YKIGVWFRQXGCECWGW656T6JMUDM7RRK6QD5CRGBB5IKMZXTQ, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.324215282 (26h13m14.107555381s) [noise=true , address=212.102.52.214:4349 , piece_id=HH47IVRCTHTDW4RHMJTEXGNB4ZK62NU6LDP5AS4FVXWPFINGHFNQ, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:29.390980339 (26h13m15.174320438s) [upload=25083]: Next returned nil segment
May 26 04:17:29.390993564 (26h13m15.174333663s) [upload=25083]: Waiting for error group managing segments...
May 26 04:17:29.673854379 (26h13m15.457194479s) [noise=true , address=212.102.52.215:4381 , piece_id=5UZ45SYYXIOMUYBHLRRP6NECZCQ5JW2RIK5MQIV2LMYQBVOON4ZA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.674402245 (26h13m15.457742344s) [noise=true , address=212.102.52.217:4433 , piece_id=FGN6M3NBSA3ZOLHYPT3THZV4THHNFYSI2R6BDB4AWSQZRCZ2M7TQ, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.677827470 (26h13m15.461167569s) [noise=true , address=212.102.52.213:4344 , piece_id=UEGMZ7I63LXBAKRWSFQDHBZBXPCHG2NMZSGW4PFVHFQORZLRMQYA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.678368041 (26h13m15.46170814s) [noise=true , address=212.102.52.215:4379 , piece_id=FBM4P5FPAP6OCVSHW5QYJOJWRXCO63HY2I6K3EYZKL6BRIUMME5Q, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.680215035 (26h13m15.463555134s) [noise=true , address=212.102.52.216:4402 , piece_id=VNTORDVWCQPSO5C2QUQUI76CBCTFB7TIQSEL7MWCCYRJ22T4WIBA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.682230351 (26h13m15.46557045s) [noise=true , address=212.102.52.217:4420 , piece_id=PQM3W3VAO5T2BBPQEREGSHGZ3EYBGJ7LFORGCKUY6NTUP2NNE5PA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.685255182 (26h13m15.468595281s) [noise=true , address=212.102.52.216:4405 , piece_id=65IGD4H7EMWEDYW2TFH643JJ5QW2BKHVFZA6D77MCHRYA6T6ESGQ, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.686472445 (26h13m15.469812544s) [noise=true , address=212.102.52.214:4362 , piece_id=LB7JGSWGNGU7BQX4FZQLVA2IUDQJT7ZP5PFXOQV4NLYXH7JAFRHA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.689410080 (26h13m15.472750179s) [noise=true , address=212.102.52.217:4436 , piece_id=SWQCVL47FBVYZFZIQVYNAD4NYDQLBBJL3S2EYVNHGS2Y54RRAN6A, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.689953157 (26h13m15.473293256s) [noise=true , address=212.102.52.214:4355 , piece_id=RKSUNVZS66X4A7ZY7WEUQXSWOTK2H5VAR7TBDICVC7MGVHA7LTGA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.694371046 (26h13m15.477711155s) [noise=true , address=212.102.52.214:4347 , piece_id=VPYOK5SVHEPZ36OOB7PYDAAINGIOUACK4FVDZBMJLWKMWC2D4NXQ, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.698718109 (26h13m15.482058208s) [noise=true , address=212.102.52.214:4349 , piece_id=HH47IVRCTHTDW4RHMJTEXGNB4ZK62NU6LDP5AS4FVXWPFINGHFNQ, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.699290191 (26h13m15.48263029s) [noise=true , address=212.102.52.215:4389 , piece_id=2OTWYJIV6A2646HAW5TWBTSQVT25V2PBKVANLGGK5SWNVHE3WGWA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.701256312 (26h13m15.484596411s) [noise=true , address=212.102.52.216:4397 , piece_id=5UNX4TPU63Q4AG726PSLVBFWX7ZZJL766ULEHEIIU3S4LG3EUGDA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.735255313 (26h13m15.518595412s) [noise=true , address=212.102.52.213:4331 , piece_id=LTU5YKIGVWFRQXGCECWGW656T6JMUDM7RRK6QD5CRGBB5IKMZXTQ, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.738998335 (26h13m15.522338434s) [noise=true , address=212.102.52.215:4380 , piece_id=6XJ45LCPGAIOCFHI35GODZRUNRS2Z63RF6WIKFUXV5R3GMSH7YJQ, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.739008105 (26h13m15.522348204s) [seg_pos={0 0}, upload=25083]: Satisfied cohort requirements after 16 pieces
May 26 04:17:29.739201553 (26h13m15.522541652s) [noise=true , address=212.102.52.217:4423 , piece_id=2KWQV3P3UYSFMBJ5BJNIHGVNZZXSCD3DEJXUV7SYVDALJSXIJAPA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TgGtX5XRdWH1qn6V5fZpQ8cY5oA2NyZx1AM17XANmr99AG9oh): protocol: expected piece hash; context canceled; EOF
May 26 04:17:29.739208446 (26h13m15.522548545s) [noise=true , address=212.102.52.217:4423 , piece_id=2KWQV3P3UYSFMBJ5BJNIHGVNZZXSCD3DEJXUV7SYVDALJSXIJAPA, seg_pos={0 0}, upload=25083]: Long tail cancellation due to optimal threshold
May 26 04:17:29.739277749 (26h13m15.522617858s) [noise=true , address=212.102.52.213:4343 , piece_id=AUT3LEUM7MPZIL33QGT47V7QOOZTHR3A2W2S6EMKJBY4B3AUA4ZA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129SbAN2aTSXzo8JLELV2ssdkmT5k7kbTcvxSZE4sFsVaKw1Aok): protocol: expected piece hash; context canceled; EOF
May 26 04:17:29.739288459 (26h13m15.522628598s) [noise=true , address=212.102.52.213:4343 , piece_id=AUT3LEUM7MPZIL33QGT47V7QOOZTHR3A2W2S6EMKJBY4B3AUA4ZA, seg_pos={0 0}, upload=25083]: Long tail cancellation due to optimal threshold
May 26 04:17:29.739324538 (26h13m15.522664637s) [noise=true , address=212.102.52.216:4408 , piece_id=23VTNWQ4FKNZ4O552CZQERQ5KMMHSCP4Y46PNSMWU4HGHSDFLIOA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BsdeZy1odVCAgcjxU6KdveXhUUTJKviRrrHLNCUuVergW6zVk): protocol: expected piece hash; context canceled; EOF
May 26 04:17:29.739330078 (26h13m15.522670177s) [noise=true , address=212.102.52.216:4408 , piece_id=23VTNWQ4FKNZ4O552CZQERQ5KMMHSCP4Y46PNSMWU4HGHSDFLIOA, seg_pos={0 0}, upload=25083]: Long tail cancellation due to optimal threshold
May 26 04:17:29.739420791 (26h13m15.52276089s) [noise=true , address=212.102.52.213:4333 , piece_id=O4MDU2BWFAOHKWNQ3KS24KNT6VYDQ4MGLBGCWW4WZZTPMVBT4OCQ, seg_pos={0 0}, upload=25083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Gm3eZxZPxPeHqLQ32kpXMpCgnwRoP8epBYtiFF1EH3U9LEazY): protocol: expected piece hash; context canceled; EOF
May 26 04:17:29.739426021 (26h13m15.52276612s) [noise=true , address=212.102.52.213:4333 , piece_id=O4MDU2BWFAOHKWNQ3KS24KNT6VYDQ4MGLBGCWW4WZZTPMVBT4OCQ, seg_pos={0 0}, upload=25083]: Long tail cancellation due to optimal threshold
May 26 04:17:29.739475536 (26h13m15.522815635s) [seg_pos={0 0}, upload=25083]: Done waiting for segment. successful: 16 optimal: 16 errs: <nil>
May 26 04:17:29.739535340 (26h13m15.522875439s) [upload=25083]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 26 04:17:29.739552654 (26h13m15.522892753s) [upload=25083]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 26 04:17:39.024892360 (26h13m24.808232459s) [upload=25083]: Done uploading segments...
May 26 04:17:39.024959289 (26h13m24.808299388s) [upload=25083]: Upload finished. err: metaclient: context deadline exceeded
Duration: 4.062552885s
core
uplink
uk-select-1
uk1-placement-17
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [.......X.....................................X..............] duration:
3.818745134s trace:
https://jaeger.dev.storj.io/trace/2f0b656030b9f519
FAIL (2026-05-26 04:33:01 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.818766577s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [........X.....................................X.............] duration:
3.814070705s trace:
https://jaeger.dev.storj.io/trace/2f0b656030b9f519
FAIL (2026-05-26 04:32:32 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.814309332s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)antiblip:
3 out of 60 is failed (threshold=3) [..........X..........X................X.....................] duration:
4.73467605s trace:
https://jaeger.dev.storj.io/trace/20840979a8f7088f
FAIL (2026-05-26 04:32:19 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
25083]: Uploading piece...
May 26 04:17:28.318973560 (26h13m14.10231366s) [noise=true , address=212.102.52.217:4420 , piece_id=PQM3W3VAO5T2BBPQEREGSHGZ3EYBGJ7LFORGCKUY6NTUP2NNE5PA, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.319026442 (26h13m14.102366541s) [noise=true , address=212.102.52.216:4397 , piece_id=5UNX4TPU63Q4AG726PSLVBFWX7ZZJL766ULEHEIIU3S4LG3EUGDA, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.319245079 (26h13m14.102585178s) [noise=true , address=212.102.52.213:4343 , piece_id=AUT3LEUM7MPZIL33QGT47V7QOOZTHR3A2W2S6EMKJBY4B3AUA4ZA, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.319586399 (26h13m14.102926499s) [noise=true , address=212.102.52.216:4402 , piece_id=VNTORDVWCQPSO5C2QUQUI76CBCTFB7TIQSEL7MWCCYRJ22T4WIBA, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.319879910 (26h13m14.103220009s) [noise=true , address=212.102.52.217:4436 , piece_id=SWQCVL47FBVYZFZIQVYNAD4NYDQLBBJL3S2EYVNHGS2Y54RRAN6A, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.320133124 (26h13m14.103473223s) [noise=true , address=212.102.52.215:4379 , piece_id=FBM4P5FPAP6OCVSHW5QYJOJWRXCO63HY2I6K3EYZKL6BRIUMME5Q, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.320357862 (26h13m14.103697961s) [noise=true , address=212.102.52.214:4362 , piece_id=LB7JGSWGNGU7BQX4FZQLVA2IUDQJT7ZP5PFXOQV4NLYXH7JAFRHA, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.320571960 (26h13m14.103912059s) [noise=true , address=212.102.52.216:4408 , piece_id=23VTNWQ4FKNZ4O552CZQERQ5KMMHSCP4Y46PNSMWU4HGHSDFLIOA, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.320588152 (26h13m14.103928251s) [noise=true , address=212.102.52.213:4344 , piece_id=UEGMZ7I63LXBAKRWSFQDHBZBXPCHG2NMZSGW4PFVHFQORZLRMQYA, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.321070803 (26h13m14.104410912s) [noise=true , address=212.102.52.217:4433 , piece_id=FGN6M3NBSA3ZOLHYPT3THZV4THHNFYSI2R6BDB4AWSQZRCZ2M7TQ, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.321436691 (26h13m14.10477679s) [noise=true , address=212.102.52.215:4380 , piece_id=6XJ45LCPGAIOCFHI35GODZRUNRS2Z63RF6WIKFUXV5R3GMSH7YJQ, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.321852404 (26h13m14.105192503s) [noise=true , address=212.102.52.213:4333 , piece_id=O4MDU2BWFAOHKWNQ3KS24KNT6VYDQ4MGLBGCWW4WZZTPMVBT4OCQ, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.322253419 (26h13m14.105593518s) [noise=true , address=212.102.52.214:4347 , piece_id=VPYOK5SVHEPZ36OOB7PYDAAINGIOUACK4FVDZBMJLWKMWC2D4NXQ, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.322652791 (26h13m14.1059929s) [noise=true , address=212.102.52.213:4331 , piece_id=LTU5YKIGVWFRQXGCECWGW656T6JMUDM7RRK6QD5CRGBB5IKMZXTQ, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.324215282 (26h13m14.107555381s) [noise=true , address=212.102.52.214:4349 , piece_id=HH47IVRCTHTDW4RHMJTEXGNB4ZK62NU6LDP5AS4FVXWPFINGHFNQ, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:29.390980339 (26h13m15.174320438s) [upload=25083]: Next returned nil segment
May 26 04:17:29.390993564 (26h13m15.174333663s) [upload=25083]: Waiting for error group managing segments...
May 26 04:17:29.673854379 (26h13m15.457194479s) [noise=true , address=212.102.52.215:4381 , piece_id=5UZ45SYYXIOMUYBHLRRP6NECZCQ5JW2RIK5MQIV2LMYQBVOON4ZA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.674402245 (26h13m15.457742344s) [noise=true , address=212.102.52.217:4433 , piece_id=FGN6M3NBSA3ZOLHYPT3THZV4THHNFYSI2R6BDB4AWSQZRCZ2M7TQ, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.677827470 (26h13m15.461167569s) [noise=true , address=212.102.52.213:4344 , piece_id=UEGMZ7I63LXBAKRWSFQDHBZBXPCHG2NMZSGW4PFVHFQORZLRMQYA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.678368041 (26h13m15.46170814s) [noise=true , address=212.102.52.215:4379 , piece_id=FBM4P5FPAP6OCVSHW5QYJOJWRXCO63HY2I6K3EYZKL6BRIUMME5Q, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.680215035 (26h13m15.463555134s) [noise=true , address=212.102.52.216:4402 , piece_id=VNTORDVWCQPSO5C2QUQUI76CBCTFB7TIQSEL7MWCCYRJ22T4WIBA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.682230351 (26h13m15.46557045s) [noise=true , address=212.102.52.217:4420 , piece_id=PQM3W3VAO5T2BBPQEREGSHGZ3EYBGJ7LFORGCKUY6NTUP2NNE5PA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.685255182 (26h13m15.468595281s) [noise=true , address=212.102.52.216:4405 , piece_id=65IGD4H7EMWEDYW2TFH643JJ5QW2BKHVFZA6D77MCHRYA6T6ESGQ, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.686472445 (26h13m15.469812544s) [noise=true , address=212.102.52.214:4362 , piece_id=LB7JGSWGNGU7BQX4FZQLVA2IUDQJT7ZP5PFXOQV4NLYXH7JAFRHA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.689410080 (26h13m15.472750179s) [noise=true , address=212.102.52.217:4436 , piece_id=SWQCVL47FBVYZFZIQVYNAD4NYDQLBBJL3S2EYVNHGS2Y54RRAN6A, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.689953157 (26h13m15.473293256s) [noise=true , address=212.102.52.214:4355 , piece_id=RKSUNVZS66X4A7ZY7WEUQXSWOTK2H5VAR7TBDICVC7MGVHA7LTGA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.694371046 (26h13m15.477711155s) [noise=true , address=212.102.52.214:4347 , piece_id=VPYOK5SVHEPZ36OOB7PYDAAINGIOUACK4FVDZBMJLWKMWC2D4NXQ, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.698718109 (26h13m15.482058208s) [noise=true , address=212.102.52.214:4349 , piece_id=HH47IVRCTHTDW4RHMJTEXGNB4ZK62NU6LDP5AS4FVXWPFINGHFNQ, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.699290191 (26h13m15.48263029s) [noise=true , address=212.102.52.215:4389 , piece_id=2OTWYJIV6A2646HAW5TWBTSQVT25V2PBKVANLGGK5SWNVHE3WGWA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.701256312 (26h13m15.484596411s) [noise=true , address=212.102.52.216:4397 , piece_id=5UNX4TPU63Q4AG726PSLVBFWX7ZZJL766ULEHEIIU3S4LG3EUGDA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.735255313 (26h13m15.518595412s) [noise=true , address=212.102.52.213:4331 , piece_id=LTU5YKIGVWFRQXGCECWGW656T6JMUDM7RRK6QD5CRGBB5IKMZXTQ, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.738998335 (26h13m15.522338434s) [noise=true , address=212.102.52.215:4380 , piece_id=6XJ45LCPGAIOCFHI35GODZRUNRS2Z63RF6WIKFUXV5R3GMSH7YJQ, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.739008105 (26h13m15.522348204s) [seg_pos={0 0}, upload=25083]: Satisfied cohort requirements after 16 pieces
May 26 04:17:29.739201553 (26h13m15.522541652s) [noise=true , address=212.102.52.217:4423 , piece_id=2KWQV3P3UYSFMBJ5BJNIHGVNZZXSCD3DEJXUV7SYVDALJSXIJAPA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TgGtX5XRdWH1qn6V5fZpQ8cY5oA2NyZx1AM17XANmr99AG9oh): protocol: expected piece hash; context canceled; EOF
May 26 04:17:29.739208446 (26h13m15.522548545s) [noise=true , address=212.102.52.217:4423 , piece_id=2KWQV3P3UYSFMBJ5BJNIHGVNZZXSCD3DEJXUV7SYVDALJSXIJAPA, seg_pos={0 0}, upload=25083]: Long tail cancellation due to optimal threshold
May 26 04:17:29.739277749 (26h13m15.522617858s) [noise=true , address=212.102.52.213:4343 , piece_id=AUT3LEUM7MPZIL33QGT47V7QOOZTHR3A2W2S6EMKJBY4B3AUA4ZA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129SbAN2aTSXzo8JLELV2ssdkmT5k7kbTcvxSZE4sFsVaKw1Aok): protocol: expected piece hash; context canceled; EOF
May 26 04:17:29.739288459 (26h13m15.522628598s) [noise=true , address=212.102.52.213:4343 , piece_id=AUT3LEUM7MPZIL33QGT47V7QOOZTHR3A2W2S6EMKJBY4B3AUA4ZA, seg_pos={0 0}, upload=25083]: Long tail cancellation due to optimal threshold
May 26 04:17:29.739324538 (26h13m15.522664637s) [noise=true , address=212.102.52.216:4408 , piece_id=23VTNWQ4FKNZ4O552CZQERQ5KMMHSCP4Y46PNSMWU4HGHSDFLIOA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BsdeZy1odVCAgcjxU6KdveXhUUTJKviRrrHLNCUuVergW6zVk): protocol: expected piece hash; context canceled; EOF
May 26 04:17:29.739330078 (26h13m15.522670177s) [noise=true , address=212.102.52.216:4408 , piece_id=23VTNWQ4FKNZ4O552CZQERQ5KMMHSCP4Y46PNSMWU4HGHSDFLIOA, seg_pos={0 0}, upload=25083]: Long tail cancellation due to optimal threshold
May 26 04:17:29.739420791 (26h13m15.52276089s) [noise=true , address=212.102.52.213:4333 , piece_id=O4MDU2BWFAOHKWNQ3KS24KNT6VYDQ4MGLBGCWW4WZZTPMVBT4OCQ, seg_pos={0 0}, upload=25083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Gm3eZxZPxPeHqLQ32kpXMpCgnwRoP8epBYtiFF1EH3U9LEazY): protocol: expected piece hash; context canceled; EOF
May 26 04:17:29.739426021 (26h13m15.52276612s) [noise=true , address=212.102.52.213:4333 , piece_id=O4MDU2BWFAOHKWNQ3KS24KNT6VYDQ4MGLBGCWW4WZZTPMVBT4OCQ, seg_pos={0 0}, upload=25083]: Long tail cancellation due to optimal threshold
May 26 04:17:29.739475536 (26h13m15.522815635s) [seg_pos={0 0}, upload=25083]: Done waiting for segment. successful: 16 optimal: 16 errs: <nil>
May 26 04:17:29.739535340 (26h13m15.522875439s) [upload=25083]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 26 04:17:29.739552654 (26h13m15.522892753s) [upload=25083]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 26 04:17:39.024892360 (26h13m24.808232459s) [upload=25083]: Done uploading segments...
May 26 04:17:39.024959289 (26h13m24.808299388s) [upload=25083]: Upload finished. err: metaclient: context deadline exceeded
Duration: 4.734700638s
core
uplink
uk-select-1
uk1-placement-17
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [.........X.....................................X............] duration:
3.83518346s trace:
https://jaeger.dev.storj.io/trace/2f0b656030b9f519
FAIL (2026-05-26 04:32:04 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.835202936s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)antiblip:
3 out of 60 is failed (threshold=3) [...........X..........X................X....................] duration:
3.950667095s trace:
https://jaeger.dev.storj.io/trace/20840979a8f7088f
FAIL (2026-05-26 04:32:02 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
25083]: Uploading piece...
May 26 04:17:28.318973560 (26h13m14.10231366s) [noise=true , address=212.102.52.217:4420 , piece_id=PQM3W3VAO5T2BBPQEREGSHGZ3EYBGJ7LFORGCKUY6NTUP2NNE5PA, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.319026442 (26h13m14.102366541s) [noise=true , address=212.102.52.216:4397 , piece_id=5UNX4TPU63Q4AG726PSLVBFWX7ZZJL766ULEHEIIU3S4LG3EUGDA, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.319245079 (26h13m14.102585178s) [noise=true , address=212.102.52.213:4343 , piece_id=AUT3LEUM7MPZIL33QGT47V7QOOZTHR3A2W2S6EMKJBY4B3AUA4ZA, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.319586399 (26h13m14.102926499s) [noise=true , address=212.102.52.216:4402 , piece_id=VNTORDVWCQPSO5C2QUQUI76CBCTFB7TIQSEL7MWCCYRJ22T4WIBA, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.319879910 (26h13m14.103220009s) [noise=true , address=212.102.52.217:4436 , piece_id=SWQCVL47FBVYZFZIQVYNAD4NYDQLBBJL3S2EYVNHGS2Y54RRAN6A, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.320133124 (26h13m14.103473223s) [noise=true , address=212.102.52.215:4379 , piece_id=FBM4P5FPAP6OCVSHW5QYJOJWRXCO63HY2I6K3EYZKL6BRIUMME5Q, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.320357862 (26h13m14.103697961s) [noise=true , address=212.102.52.214:4362 , piece_id=LB7JGSWGNGU7BQX4FZQLVA2IUDQJT7ZP5PFXOQV4NLYXH7JAFRHA, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.320571960 (26h13m14.103912059s) [noise=true , address=212.102.52.216:4408 , piece_id=23VTNWQ4FKNZ4O552CZQERQ5KMMHSCP4Y46PNSMWU4HGHSDFLIOA, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.320588152 (26h13m14.103928251s) [noise=true , address=212.102.52.213:4344 , piece_id=UEGMZ7I63LXBAKRWSFQDHBZBXPCHG2NMZSGW4PFVHFQORZLRMQYA, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.321070803 (26h13m14.104410912s) [noise=true , address=212.102.52.217:4433 , piece_id=FGN6M3NBSA3ZOLHYPT3THZV4THHNFYSI2R6BDB4AWSQZRCZ2M7TQ, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.321436691 (26h13m14.10477679s) [noise=true , address=212.102.52.215:4380 , piece_id=6XJ45LCPGAIOCFHI35GODZRUNRS2Z63RF6WIKFUXV5R3GMSH7YJQ, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.321852404 (26h13m14.105192503s) [noise=true , address=212.102.52.213:4333 , piece_id=O4MDU2BWFAOHKWNQ3KS24KNT6VYDQ4MGLBGCWW4WZZTPMVBT4OCQ, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.322253419 (26h13m14.105593518s) [noise=true , address=212.102.52.214:4347 , piece_id=VPYOK5SVHEPZ36OOB7PYDAAINGIOUACK4FVDZBMJLWKMWC2D4NXQ, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.322652791 (26h13m14.1059929s) [noise=true , address=212.102.52.213:4331 , piece_id=LTU5YKIGVWFRQXGCECWGW656T6JMUDM7RRK6QD5CRGBB5IKMZXTQ, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.324215282 (26h13m14.107555381s) [noise=true , address=212.102.52.214:4349 , piece_id=HH47IVRCTHTDW4RHMJTEXGNB4ZK62NU6LDP5AS4FVXWPFINGHFNQ, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:29.390980339 (26h13m15.174320438s) [upload=25083]: Next returned nil segment
May 26 04:17:29.390993564 (26h13m15.174333663s) [upload=25083]: Waiting for error group managing segments...
May 26 04:17:29.673854379 (26h13m15.457194479s) [noise=true , address=212.102.52.215:4381 , piece_id=5UZ45SYYXIOMUYBHLRRP6NECZCQ5JW2RIK5MQIV2LMYQBVOON4ZA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.674402245 (26h13m15.457742344s) [noise=true , address=212.102.52.217:4433 , piece_id=FGN6M3NBSA3ZOLHYPT3THZV4THHNFYSI2R6BDB4AWSQZRCZ2M7TQ, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.677827470 (26h13m15.461167569s) [noise=true , address=212.102.52.213:4344 , piece_id=UEGMZ7I63LXBAKRWSFQDHBZBXPCHG2NMZSGW4PFVHFQORZLRMQYA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.678368041 (26h13m15.46170814s) [noise=true , address=212.102.52.215:4379 , piece_id=FBM4P5FPAP6OCVSHW5QYJOJWRXCO63HY2I6K3EYZKL6BRIUMME5Q, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.680215035 (26h13m15.463555134s) [noise=true , address=212.102.52.216:4402 , piece_id=VNTORDVWCQPSO5C2QUQUI76CBCTFB7TIQSEL7MWCCYRJ22T4WIBA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.682230351 (26h13m15.46557045s) [noise=true , address=212.102.52.217:4420 , piece_id=PQM3W3VAO5T2BBPQEREGSHGZ3EYBGJ7LFORGCKUY6NTUP2NNE5PA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.685255182 (26h13m15.468595281s) [noise=true , address=212.102.52.216:4405 , piece_id=65IGD4H7EMWEDYW2TFH643JJ5QW2BKHVFZA6D77MCHRYA6T6ESGQ, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.686472445 (26h13m15.469812544s) [noise=true , address=212.102.52.214:4362 , piece_id=LB7JGSWGNGU7BQX4FZQLVA2IUDQJT7ZP5PFXOQV4NLYXH7JAFRHA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.689410080 (26h13m15.472750179s) [noise=true , address=212.102.52.217:4436 , piece_id=SWQCVL47FBVYZFZIQVYNAD4NYDQLBBJL3S2EYVNHGS2Y54RRAN6A, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.689953157 (26h13m15.473293256s) [noise=true , address=212.102.52.214:4355 , piece_id=RKSUNVZS66X4A7ZY7WEUQXSWOTK2H5VAR7TBDICVC7MGVHA7LTGA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.694371046 (26h13m15.477711155s) [noise=true , address=212.102.52.214:4347 , piece_id=VPYOK5SVHEPZ36OOB7PYDAAINGIOUACK4FVDZBMJLWKMWC2D4NXQ, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.698718109 (26h13m15.482058208s) [noise=true , address=212.102.52.214:4349 , piece_id=HH47IVRCTHTDW4RHMJTEXGNB4ZK62NU6LDP5AS4FVXWPFINGHFNQ, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.699290191 (26h13m15.48263029s) [noise=true , address=212.102.52.215:4389 , piece_id=2OTWYJIV6A2646HAW5TWBTSQVT25V2PBKVANLGGK5SWNVHE3WGWA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.701256312 (26h13m15.484596411s) [noise=true , address=212.102.52.216:4397 , piece_id=5UNX4TPU63Q4AG726PSLVBFWX7ZZJL766ULEHEIIU3S4LG3EUGDA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.735255313 (26h13m15.518595412s) [noise=true , address=212.102.52.213:4331 , piece_id=LTU5YKIGVWFRQXGCECWGW656T6JMUDM7RRK6QD5CRGBB5IKMZXTQ, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.738998335 (26h13m15.522338434s) [noise=true , address=212.102.52.215:4380 , piece_id=6XJ45LCPGAIOCFHI35GODZRUNRS2Z63RF6WIKFUXV5R3GMSH7YJQ, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.739008105 (26h13m15.522348204s) [seg_pos={0 0}, upload=25083]: Satisfied cohort requirements after 16 pieces
May 26 04:17:29.739201553 (26h13m15.522541652s) [noise=true , address=212.102.52.217:4423 , piece_id=2KWQV3P3UYSFMBJ5BJNIHGVNZZXSCD3DEJXUV7SYVDALJSXIJAPA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TgGtX5XRdWH1qn6V5fZpQ8cY5oA2NyZx1AM17XANmr99AG9oh): protocol: expected piece hash; context canceled; EOF
May 26 04:17:29.739208446 (26h13m15.522548545s) [noise=true , address=212.102.52.217:4423 , piece_id=2KWQV3P3UYSFMBJ5BJNIHGVNZZXSCD3DEJXUV7SYVDALJSXIJAPA, seg_pos={0 0}, upload=25083]: Long tail cancellation due to optimal threshold
May 26 04:17:29.739277749 (26h13m15.522617858s) [noise=true , address=212.102.52.213:4343 , piece_id=AUT3LEUM7MPZIL33QGT47V7QOOZTHR3A2W2S6EMKJBY4B3AUA4ZA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129SbAN2aTSXzo8JLELV2ssdkmT5k7kbTcvxSZE4sFsVaKw1Aok): protocol: expected piece hash; context canceled; EOF
May 26 04:17:29.739288459 (26h13m15.522628598s) [noise=true , address=212.102.52.213:4343 , piece_id=AUT3LEUM7MPZIL33QGT47V7QOOZTHR3A2W2S6EMKJBY4B3AUA4ZA, seg_pos={0 0}, upload=25083]: Long tail cancellation due to optimal threshold
May 26 04:17:29.739324538 (26h13m15.522664637s) [noise=true , address=212.102.52.216:4408 , piece_id=23VTNWQ4FKNZ4O552CZQERQ5KMMHSCP4Y46PNSMWU4HGHSDFLIOA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BsdeZy1odVCAgcjxU6KdveXhUUTJKviRrrHLNCUuVergW6zVk): protocol: expected piece hash; context canceled; EOF
May 26 04:17:29.739330078 (26h13m15.522670177s) [noise=true , address=212.102.52.216:4408 , piece_id=23VTNWQ4FKNZ4O552CZQERQ5KMMHSCP4Y46PNSMWU4HGHSDFLIOA, seg_pos={0 0}, upload=25083]: Long tail cancellation due to optimal threshold
May 26 04:17:29.739420791 (26h13m15.52276089s) [noise=true , address=212.102.52.213:4333 , piece_id=O4MDU2BWFAOHKWNQ3KS24KNT6VYDQ4MGLBGCWW4WZZTPMVBT4OCQ, seg_pos={0 0}, upload=25083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Gm3eZxZPxPeHqLQ32kpXMpCgnwRoP8epBYtiFF1EH3U9LEazY): protocol: expected piece hash; context canceled; EOF
May 26 04:17:29.739426021 (26h13m15.52276612s) [noise=true , address=212.102.52.213:4333 , piece_id=O4MDU2BWFAOHKWNQ3KS24KNT6VYDQ4MGLBGCWW4WZZTPMVBT4OCQ, seg_pos={0 0}, upload=25083]: Long tail cancellation due to optimal threshold
May 26 04:17:29.739475536 (26h13m15.522815635s) [seg_pos={0 0}, upload=25083]: Done waiting for segment. successful: 16 optimal: 16 errs: <nil>
May 26 04:17:29.739535340 (26h13m15.522875439s) [upload=25083]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 26 04:17:29.739552654 (26h13m15.522892753s) [upload=25083]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 26 04:17:39.024892360 (26h13m24.808232459s) [upload=25083]: Done uploading segments...
May 26 04:17:39.024959289 (26h13m24.808299388s) [upload=25083]: Upload finished. err: metaclient: context deadline exceeded
Duration: 3.950689939s
core
uplink
uk-select-1
uk1-placement-17
|
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)antiblip:
3 out of 60 is failed (threshold=3) [............X..........X................X...................] duration:
4.154589083s trace:
https://jaeger.dev.storj.io/trace/20840979a8f7088f
FAIL (2026-05-26 04:31:39 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
25083]: Uploading piece...
May 26 04:17:28.318973560 (26h13m14.10231366s) [noise=true , address=212.102.52.217:4420 , piece_id=PQM3W3VAO5T2BBPQEREGSHGZ3EYBGJ7LFORGCKUY6NTUP2NNE5PA, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.319026442 (26h13m14.102366541s) [noise=true , address=212.102.52.216:4397 , piece_id=5UNX4TPU63Q4AG726PSLVBFWX7ZZJL766ULEHEIIU3S4LG3EUGDA, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.319245079 (26h13m14.102585178s) [noise=true , address=212.102.52.213:4343 , piece_id=AUT3LEUM7MPZIL33QGT47V7QOOZTHR3A2W2S6EMKJBY4B3AUA4ZA, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.319586399 (26h13m14.102926499s) [noise=true , address=212.102.52.216:4402 , piece_id=VNTORDVWCQPSO5C2QUQUI76CBCTFB7TIQSEL7MWCCYRJ22T4WIBA, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.319879910 (26h13m14.103220009s) [noise=true , address=212.102.52.217:4436 , piece_id=SWQCVL47FBVYZFZIQVYNAD4NYDQLBBJL3S2EYVNHGS2Y54RRAN6A, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.320133124 (26h13m14.103473223s) [noise=true , address=212.102.52.215:4379 , piece_id=FBM4P5FPAP6OCVSHW5QYJOJWRXCO63HY2I6K3EYZKL6BRIUMME5Q, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.320357862 (26h13m14.103697961s) [noise=true , address=212.102.52.214:4362 , piece_id=LB7JGSWGNGU7BQX4FZQLVA2IUDQJT7ZP5PFXOQV4NLYXH7JAFRHA, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.320571960 (26h13m14.103912059s) [noise=true , address=212.102.52.216:4408 , piece_id=23VTNWQ4FKNZ4O552CZQERQ5KMMHSCP4Y46PNSMWU4HGHSDFLIOA, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.320588152 (26h13m14.103928251s) [noise=true , address=212.102.52.213:4344 , piece_id=UEGMZ7I63LXBAKRWSFQDHBZBXPCHG2NMZSGW4PFVHFQORZLRMQYA, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.321070803 (26h13m14.104410912s) [noise=true , address=212.102.52.217:4433 , piece_id=FGN6M3NBSA3ZOLHYPT3THZV4THHNFYSI2R6BDB4AWSQZRCZ2M7TQ, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.321436691 (26h13m14.10477679s) [noise=true , address=212.102.52.215:4380 , piece_id=6XJ45LCPGAIOCFHI35GODZRUNRS2Z63RF6WIKFUXV5R3GMSH7YJQ, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.321852404 (26h13m14.105192503s) [noise=true , address=212.102.52.213:4333 , piece_id=O4MDU2BWFAOHKWNQ3KS24KNT6VYDQ4MGLBGCWW4WZZTPMVBT4OCQ, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.322253419 (26h13m14.105593518s) [noise=true , address=212.102.52.214:4347 , piece_id=VPYOK5SVHEPZ36OOB7PYDAAINGIOUACK4FVDZBMJLWKMWC2D4NXQ, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.322652791 (26h13m14.1059929s) [noise=true , address=212.102.52.213:4331 , piece_id=LTU5YKIGVWFRQXGCECWGW656T6JMUDM7RRK6QD5CRGBB5IKMZXTQ, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.324215282 (26h13m14.107555381s) [noise=true , address=212.102.52.214:4349 , piece_id=HH47IVRCTHTDW4RHMJTEXGNB4ZK62NU6LDP5AS4FVXWPFINGHFNQ, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:29.390980339 (26h13m15.174320438s) [upload=25083]: Next returned nil segment
May 26 04:17:29.390993564 (26h13m15.174333663s) [upload=25083]: Waiting for error group managing segments...
May 26 04:17:29.673854379 (26h13m15.457194479s) [noise=true , address=212.102.52.215:4381 , piece_id=5UZ45SYYXIOMUYBHLRRP6NECZCQ5JW2RIK5MQIV2LMYQBVOON4ZA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.674402245 (26h13m15.457742344s) [noise=true , address=212.102.52.217:4433 , piece_id=FGN6M3NBSA3ZOLHYPT3THZV4THHNFYSI2R6BDB4AWSQZRCZ2M7TQ, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.677827470 (26h13m15.461167569s) [noise=true , address=212.102.52.213:4344 , piece_id=UEGMZ7I63LXBAKRWSFQDHBZBXPCHG2NMZSGW4PFVHFQORZLRMQYA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.678368041 (26h13m15.46170814s) [noise=true , address=212.102.52.215:4379 , piece_id=FBM4P5FPAP6OCVSHW5QYJOJWRXCO63HY2I6K3EYZKL6BRIUMME5Q, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.680215035 (26h13m15.463555134s) [noise=true , address=212.102.52.216:4402 , piece_id=VNTORDVWCQPSO5C2QUQUI76CBCTFB7TIQSEL7MWCCYRJ22T4WIBA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.682230351 (26h13m15.46557045s) [noise=true , address=212.102.52.217:4420 , piece_id=PQM3W3VAO5T2BBPQEREGSHGZ3EYBGJ7LFORGCKUY6NTUP2NNE5PA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.685255182 (26h13m15.468595281s) [noise=true , address=212.102.52.216:4405 , piece_id=65IGD4H7EMWEDYW2TFH643JJ5QW2BKHVFZA6D77MCHRYA6T6ESGQ, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.686472445 (26h13m15.469812544s) [noise=true , address=212.102.52.214:4362 , piece_id=LB7JGSWGNGU7BQX4FZQLVA2IUDQJT7ZP5PFXOQV4NLYXH7JAFRHA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.689410080 (26h13m15.472750179s) [noise=true , address=212.102.52.217:4436 , piece_id=SWQCVL47FBVYZFZIQVYNAD4NYDQLBBJL3S2EYVNHGS2Y54RRAN6A, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.689953157 (26h13m15.473293256s) [noise=true , address=212.102.52.214:4355 , piece_id=RKSUNVZS66X4A7ZY7WEUQXSWOTK2H5VAR7TBDICVC7MGVHA7LTGA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.694371046 (26h13m15.477711155s) [noise=true , address=212.102.52.214:4347 , piece_id=VPYOK5SVHEPZ36OOB7PYDAAINGIOUACK4FVDZBMJLWKMWC2D4NXQ, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.698718109 (26h13m15.482058208s) [noise=true , address=212.102.52.214:4349 , piece_id=HH47IVRCTHTDW4RHMJTEXGNB4ZK62NU6LDP5AS4FVXWPFINGHFNQ, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.699290191 (26h13m15.48263029s) [noise=true , address=212.102.52.215:4389 , piece_id=2OTWYJIV6A2646HAW5TWBTSQVT25V2PBKVANLGGK5SWNVHE3WGWA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.701256312 (26h13m15.484596411s) [noise=true , address=212.102.52.216:4397 , piece_id=5UNX4TPU63Q4AG726PSLVBFWX7ZZJL766ULEHEIIU3S4LG3EUGDA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.735255313 (26h13m15.518595412s) [noise=true , address=212.102.52.213:4331 , piece_id=LTU5YKIGVWFRQXGCECWGW656T6JMUDM7RRK6QD5CRGBB5IKMZXTQ, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.738998335 (26h13m15.522338434s) [noise=true , address=212.102.52.215:4380 , piece_id=6XJ45LCPGAIOCFHI35GODZRUNRS2Z63RF6WIKFUXV5R3GMSH7YJQ, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.739008105 (26h13m15.522348204s) [seg_pos={0 0}, upload=25083]: Satisfied cohort requirements after 16 pieces
May 26 04:17:29.739201553 (26h13m15.522541652s) [noise=true , address=212.102.52.217:4423 , piece_id=2KWQV3P3UYSFMBJ5BJNIHGVNZZXSCD3DEJXUV7SYVDALJSXIJAPA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TgGtX5XRdWH1qn6V5fZpQ8cY5oA2NyZx1AM17XANmr99AG9oh): protocol: expected piece hash; context canceled; EOF
May 26 04:17:29.739208446 (26h13m15.522548545s) [noise=true , address=212.102.52.217:4423 , piece_id=2KWQV3P3UYSFMBJ5BJNIHGVNZZXSCD3DEJXUV7SYVDALJSXIJAPA, seg_pos={0 0}, upload=25083]: Long tail cancellation due to optimal threshold
May 26 04:17:29.739277749 (26h13m15.522617858s) [noise=true , address=212.102.52.213:4343 , piece_id=AUT3LEUM7MPZIL33QGT47V7QOOZTHR3A2W2S6EMKJBY4B3AUA4ZA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129SbAN2aTSXzo8JLELV2ssdkmT5k7kbTcvxSZE4sFsVaKw1Aok): protocol: expected piece hash; context canceled; EOF
May 26 04:17:29.739288459 (26h13m15.522628598s) [noise=true , address=212.102.52.213:4343 , piece_id=AUT3LEUM7MPZIL33QGT47V7QOOZTHR3A2W2S6EMKJBY4B3AUA4ZA, seg_pos={0 0}, upload=25083]: Long tail cancellation due to optimal threshold
May 26 04:17:29.739324538 (26h13m15.522664637s) [noise=true , address=212.102.52.216:4408 , piece_id=23VTNWQ4FKNZ4O552CZQERQ5KMMHSCP4Y46PNSMWU4HGHSDFLIOA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BsdeZy1odVCAgcjxU6KdveXhUUTJKviRrrHLNCUuVergW6zVk): protocol: expected piece hash; context canceled; EOF
May 26 04:17:29.739330078 (26h13m15.522670177s) [noise=true , address=212.102.52.216:4408 , piece_id=23VTNWQ4FKNZ4O552CZQERQ5KMMHSCP4Y46PNSMWU4HGHSDFLIOA, seg_pos={0 0}, upload=25083]: Long tail cancellation due to optimal threshold
May 26 04:17:29.739420791 (26h13m15.52276089s) [noise=true , address=212.102.52.213:4333 , piece_id=O4MDU2BWFAOHKWNQ3KS24KNT6VYDQ4MGLBGCWW4WZZTPMVBT4OCQ, seg_pos={0 0}, upload=25083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Gm3eZxZPxPeHqLQ32kpXMpCgnwRoP8epBYtiFF1EH3U9LEazY): protocol: expected piece hash; context canceled; EOF
May 26 04:17:29.739426021 (26h13m15.52276612s) [noise=true , address=212.102.52.213:4333 , piece_id=O4MDU2BWFAOHKWNQ3KS24KNT6VYDQ4MGLBGCWW4WZZTPMVBT4OCQ, seg_pos={0 0}, upload=25083]: Long tail cancellation due to optimal threshold
May 26 04:17:29.739475536 (26h13m15.522815635s) [seg_pos={0 0}, upload=25083]: Done waiting for segment. successful: 16 optimal: 16 errs: <nil>
May 26 04:17:29.739535340 (26h13m15.522875439s) [upload=25083]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 26 04:17:29.739552654 (26h13m15.522892753s) [upload=25083]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 26 04:17:39.024892360 (26h13m24.808232459s) [upload=25083]: Done uploading segments...
May 26 04:17:39.024959289 (26h13m24.808299388s) [upload=25083]: Upload finished. err: metaclient: context deadline exceeded
Duration: 4.154615191s
core
uplink
uk-select-1
uk1-placement-17
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [..........X.....................................X...........] duration:
3.722592003s trace:
https://jaeger.dev.storj.io/trace/2f0b656030b9f519
FAIL (2026-05-26 04:30:55 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.722602072s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)antiblip:
4 out of 60 is failed (threshold=3) [X............X..........X................X..................] duration:
4.186335515s trace:
https://jaeger.dev.storj.io/trace/20840979a8f7088f
FAIL (2026-05-26 04:30:42 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
25083]: Uploading piece...
May 26 04:17:28.318973560 (26h13m14.10231366s) [noise=true , address=212.102.52.217:4420 , piece_id=PQM3W3VAO5T2BBPQEREGSHGZ3EYBGJ7LFORGCKUY6NTUP2NNE5PA, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.319026442 (26h13m14.102366541s) [noise=true , address=212.102.52.216:4397 , piece_id=5UNX4TPU63Q4AG726PSLVBFWX7ZZJL766ULEHEIIU3S4LG3EUGDA, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.319245079 (26h13m14.102585178s) [noise=true , address=212.102.52.213:4343 , piece_id=AUT3LEUM7MPZIL33QGT47V7QOOZTHR3A2W2S6EMKJBY4B3AUA4ZA, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.319586399 (26h13m14.102926499s) [noise=true , address=212.102.52.216:4402 , piece_id=VNTORDVWCQPSO5C2QUQUI76CBCTFB7TIQSEL7MWCCYRJ22T4WIBA, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.319879910 (26h13m14.103220009s) [noise=true , address=212.102.52.217:4436 , piece_id=SWQCVL47FBVYZFZIQVYNAD4NYDQLBBJL3S2EYVNHGS2Y54RRAN6A, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.320133124 (26h13m14.103473223s) [noise=true , address=212.102.52.215:4379 , piece_id=FBM4P5FPAP6OCVSHW5QYJOJWRXCO63HY2I6K3EYZKL6BRIUMME5Q, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.320357862 (26h13m14.103697961s) [noise=true , address=212.102.52.214:4362 , piece_id=LB7JGSWGNGU7BQX4FZQLVA2IUDQJT7ZP5PFXOQV4NLYXH7JAFRHA, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.320571960 (26h13m14.103912059s) [noise=true , address=212.102.52.216:4408 , piece_id=23VTNWQ4FKNZ4O552CZQERQ5KMMHSCP4Y46PNSMWU4HGHSDFLIOA, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.320588152 (26h13m14.103928251s) [noise=true , address=212.102.52.213:4344 , piece_id=UEGMZ7I63LXBAKRWSFQDHBZBXPCHG2NMZSGW4PFVHFQORZLRMQYA, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.321070803 (26h13m14.104410912s) [noise=true , address=212.102.52.217:4433 , piece_id=FGN6M3NBSA3ZOLHYPT3THZV4THHNFYSI2R6BDB4AWSQZRCZ2M7TQ, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.321436691 (26h13m14.10477679s) [noise=true , address=212.102.52.215:4380 , piece_id=6XJ45LCPGAIOCFHI35GODZRUNRS2Z63RF6WIKFUXV5R3GMSH7YJQ, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.321852404 (26h13m14.105192503s) [noise=true , address=212.102.52.213:4333 , piece_id=O4MDU2BWFAOHKWNQ3KS24KNT6VYDQ4MGLBGCWW4WZZTPMVBT4OCQ, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.322253419 (26h13m14.105593518s) [noise=true , address=212.102.52.214:4347 , piece_id=VPYOK5SVHEPZ36OOB7PYDAAINGIOUACK4FVDZBMJLWKMWC2D4NXQ, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.322652791 (26h13m14.1059929s) [noise=true , address=212.102.52.213:4331 , piece_id=LTU5YKIGVWFRQXGCECWGW656T6JMUDM7RRK6QD5CRGBB5IKMZXTQ, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.324215282 (26h13m14.107555381s) [noise=true , address=212.102.52.214:4349 , piece_id=HH47IVRCTHTDW4RHMJTEXGNB4ZK62NU6LDP5AS4FVXWPFINGHFNQ, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:29.390980339 (26h13m15.174320438s) [upload=25083]: Next returned nil segment
May 26 04:17:29.390993564 (26h13m15.174333663s) [upload=25083]: Waiting for error group managing segments...
May 26 04:17:29.673854379 (26h13m15.457194479s) [noise=true , address=212.102.52.215:4381 , piece_id=5UZ45SYYXIOMUYBHLRRP6NECZCQ5JW2RIK5MQIV2LMYQBVOON4ZA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.674402245 (26h13m15.457742344s) [noise=true , address=212.102.52.217:4433 , piece_id=FGN6M3NBSA3ZOLHYPT3THZV4THHNFYSI2R6BDB4AWSQZRCZ2M7TQ, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.677827470 (26h13m15.461167569s) [noise=true , address=212.102.52.213:4344 , piece_id=UEGMZ7I63LXBAKRWSFQDHBZBXPCHG2NMZSGW4PFVHFQORZLRMQYA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.678368041 (26h13m15.46170814s) [noise=true , address=212.102.52.215:4379 , piece_id=FBM4P5FPAP6OCVSHW5QYJOJWRXCO63HY2I6K3EYZKL6BRIUMME5Q, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.680215035 (26h13m15.463555134s) [noise=true , address=212.102.52.216:4402 , piece_id=VNTORDVWCQPSO5C2QUQUI76CBCTFB7TIQSEL7MWCCYRJ22T4WIBA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.682230351 (26h13m15.46557045s) [noise=true , address=212.102.52.217:4420 , piece_id=PQM3W3VAO5T2BBPQEREGSHGZ3EYBGJ7LFORGCKUY6NTUP2NNE5PA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.685255182 (26h13m15.468595281s) [noise=true , address=212.102.52.216:4405 , piece_id=65IGD4H7EMWEDYW2TFH643JJ5QW2BKHVFZA6D77MCHRYA6T6ESGQ, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.686472445 (26h13m15.469812544s) [noise=true , address=212.102.52.214:4362 , piece_id=LB7JGSWGNGU7BQX4FZQLVA2IUDQJT7ZP5PFXOQV4NLYXH7JAFRHA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.689410080 (26h13m15.472750179s) [noise=true , address=212.102.52.217:4436 , piece_id=SWQCVL47FBVYZFZIQVYNAD4NYDQLBBJL3S2EYVNHGS2Y54RRAN6A, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.689953157 (26h13m15.473293256s) [noise=true , address=212.102.52.214:4355 , piece_id=RKSUNVZS66X4A7ZY7WEUQXSWOTK2H5VAR7TBDICVC7MGVHA7LTGA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.694371046 (26h13m15.477711155s) [noise=true , address=212.102.52.214:4347 , piece_id=VPYOK5SVHEPZ36OOB7PYDAAINGIOUACK4FVDZBMJLWKMWC2D4NXQ, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.698718109 (26h13m15.482058208s) [noise=true , address=212.102.52.214:4349 , piece_id=HH47IVRCTHTDW4RHMJTEXGNB4ZK62NU6LDP5AS4FVXWPFINGHFNQ, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.699290191 (26h13m15.48263029s) [noise=true , address=212.102.52.215:4389 , piece_id=2OTWYJIV6A2646HAW5TWBTSQVT25V2PBKVANLGGK5SWNVHE3WGWA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.701256312 (26h13m15.484596411s) [noise=true , address=212.102.52.216:4397 , piece_id=5UNX4TPU63Q4AG726PSLVBFWX7ZZJL766ULEHEIIU3S4LG3EUGDA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.735255313 (26h13m15.518595412s) [noise=true , address=212.102.52.213:4331 , piece_id=LTU5YKIGVWFRQXGCECWGW656T6JMUDM7RRK6QD5CRGBB5IKMZXTQ, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.738998335 (26h13m15.522338434s) [noise=true , address=212.102.52.215:4380 , piece_id=6XJ45LCPGAIOCFHI35GODZRUNRS2Z63RF6WIKFUXV5R3GMSH7YJQ, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.739008105 (26h13m15.522348204s) [seg_pos={0 0}, upload=25083]: Satisfied cohort requirements after 16 pieces
May 26 04:17:29.739201553 (26h13m15.522541652s) [noise=true , address=212.102.52.217:4423 , piece_id=2KWQV3P3UYSFMBJ5BJNIHGVNZZXSCD3DEJXUV7SYVDALJSXIJAPA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TgGtX5XRdWH1qn6V5fZpQ8cY5oA2NyZx1AM17XANmr99AG9oh): protocol: expected piece hash; context canceled; EOF
May 26 04:17:29.739208446 (26h13m15.522548545s) [noise=true , address=212.102.52.217:4423 , piece_id=2KWQV3P3UYSFMBJ5BJNIHGVNZZXSCD3DEJXUV7SYVDALJSXIJAPA, seg_pos={0 0}, upload=25083]: Long tail cancellation due to optimal threshold
May 26 04:17:29.739277749 (26h13m15.522617858s) [noise=true , address=212.102.52.213:4343 , piece_id=AUT3LEUM7MPZIL33QGT47V7QOOZTHR3A2W2S6EMKJBY4B3AUA4ZA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129SbAN2aTSXzo8JLELV2ssdkmT5k7kbTcvxSZE4sFsVaKw1Aok): protocol: expected piece hash; context canceled; EOF
May 26 04:17:29.739288459 (26h13m15.522628598s) [noise=true , address=212.102.52.213:4343 , piece_id=AUT3LEUM7MPZIL33QGT47V7QOOZTHR3A2W2S6EMKJBY4B3AUA4ZA, seg_pos={0 0}, upload=25083]: Long tail cancellation due to optimal threshold
May 26 04:17:29.739324538 (26h13m15.522664637s) [noise=true , address=212.102.52.216:4408 , piece_id=23VTNWQ4FKNZ4O552CZQERQ5KMMHSCP4Y46PNSMWU4HGHSDFLIOA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BsdeZy1odVCAgcjxU6KdveXhUUTJKviRrrHLNCUuVergW6zVk): protocol: expected piece hash; context canceled; EOF
May 26 04:17:29.739330078 (26h13m15.522670177s) [noise=true , address=212.102.52.216:4408 , piece_id=23VTNWQ4FKNZ4O552CZQERQ5KMMHSCP4Y46PNSMWU4HGHSDFLIOA, seg_pos={0 0}, upload=25083]: Long tail cancellation due to optimal threshold
May 26 04:17:29.739420791 (26h13m15.52276089s) [noise=true , address=212.102.52.213:4333 , piece_id=O4MDU2BWFAOHKWNQ3KS24KNT6VYDQ4MGLBGCWW4WZZTPMVBT4OCQ, seg_pos={0 0}, upload=25083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Gm3eZxZPxPeHqLQ32kpXMpCgnwRoP8epBYtiFF1EH3U9LEazY): protocol: expected piece hash; context canceled; EOF
May 26 04:17:29.739426021 (26h13m15.52276612s) [noise=true , address=212.102.52.213:4333 , piece_id=O4MDU2BWFAOHKWNQ3KS24KNT6VYDQ4MGLBGCWW4WZZTPMVBT4OCQ, seg_pos={0 0}, upload=25083]: Long tail cancellation due to optimal threshold
May 26 04:17:29.739475536 (26h13m15.522815635s) [seg_pos={0 0}, upload=25083]: Done waiting for segment. successful: 16 optimal: 16 errs: <nil>
May 26 04:17:29.739535340 (26h13m15.522875439s) [upload=25083]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 26 04:17:29.739552654 (26h13m15.522892753s) [upload=25083]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 26 04:17:39.024892360 (26h13m24.808232459s) [upload=25083]: Done uploading segments...
May 26 04:17:39.024959289 (26h13m24.808299388s) [upload=25083]: Upload finished. err: metaclient: context deadline exceeded
Duration: 4.186356324s
core
uplink
uk-select-1
uk1-placement-17
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [...........X.....................................X..........] duration:
3.751088823s trace:
https://jaeger.dev.storj.io/trace/2f0b656030b9f519
FAIL (2026-05-26 04:30:25 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.751108782s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)antiblip:
4 out of 60 is failed (threshold=3) [.X............X..........X................X.................] duration:
4.347100751s trace:
https://jaeger.dev.storj.io/trace/20840979a8f7088f
FAIL (2026-05-26 04:29:38 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
25083]: Uploading piece...
May 26 04:17:28.318973560 (26h13m14.10231366s) [noise=true , address=212.102.52.217:4420 , piece_id=PQM3W3VAO5T2BBPQEREGSHGZ3EYBGJ7LFORGCKUY6NTUP2NNE5PA, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.319026442 (26h13m14.102366541s) [noise=true , address=212.102.52.216:4397 , piece_id=5UNX4TPU63Q4AG726PSLVBFWX7ZZJL766ULEHEIIU3S4LG3EUGDA, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.319245079 (26h13m14.102585178s) [noise=true , address=212.102.52.213:4343 , piece_id=AUT3LEUM7MPZIL33QGT47V7QOOZTHR3A2W2S6EMKJBY4B3AUA4ZA, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.319586399 (26h13m14.102926499s) [noise=true , address=212.102.52.216:4402 , piece_id=VNTORDVWCQPSO5C2QUQUI76CBCTFB7TIQSEL7MWCCYRJ22T4WIBA, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.319879910 (26h13m14.103220009s) [noise=true , address=212.102.52.217:4436 , piece_id=SWQCVL47FBVYZFZIQVYNAD4NYDQLBBJL3S2EYVNHGS2Y54RRAN6A, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.320133124 (26h13m14.103473223s) [noise=true , address=212.102.52.215:4379 , piece_id=FBM4P5FPAP6OCVSHW5QYJOJWRXCO63HY2I6K3EYZKL6BRIUMME5Q, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.320357862 (26h13m14.103697961s) [noise=true , address=212.102.52.214:4362 , piece_id=LB7JGSWGNGU7BQX4FZQLVA2IUDQJT7ZP5PFXOQV4NLYXH7JAFRHA, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.320571960 (26h13m14.103912059s) [noise=true , address=212.102.52.216:4408 , piece_id=23VTNWQ4FKNZ4O552CZQERQ5KMMHSCP4Y46PNSMWU4HGHSDFLIOA, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.320588152 (26h13m14.103928251s) [noise=true , address=212.102.52.213:4344 , piece_id=UEGMZ7I63LXBAKRWSFQDHBZBXPCHG2NMZSGW4PFVHFQORZLRMQYA, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.321070803 (26h13m14.104410912s) [noise=true , address=212.102.52.217:4433 , piece_id=FGN6M3NBSA3ZOLHYPT3THZV4THHNFYSI2R6BDB4AWSQZRCZ2M7TQ, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.321436691 (26h13m14.10477679s) [noise=true , address=212.102.52.215:4380 , piece_id=6XJ45LCPGAIOCFHI35GODZRUNRS2Z63RF6WIKFUXV5R3GMSH7YJQ, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.321852404 (26h13m14.105192503s) [noise=true , address=212.102.52.213:4333 , piece_id=O4MDU2BWFAOHKWNQ3KS24KNT6VYDQ4MGLBGCWW4WZZTPMVBT4OCQ, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.322253419 (26h13m14.105593518s) [noise=true , address=212.102.52.214:4347 , piece_id=VPYOK5SVHEPZ36OOB7PYDAAINGIOUACK4FVDZBMJLWKMWC2D4NXQ, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.322652791 (26h13m14.1059929s) [noise=true , address=212.102.52.213:4331 , piece_id=LTU5YKIGVWFRQXGCECWGW656T6JMUDM7RRK6QD5CRGBB5IKMZXTQ, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.324215282 (26h13m14.107555381s) [noise=true , address=212.102.52.214:4349 , piece_id=HH47IVRCTHTDW4RHMJTEXGNB4ZK62NU6LDP5AS4FVXWPFINGHFNQ, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:29.390980339 (26h13m15.174320438s) [upload=25083]: Next returned nil segment
May 26 04:17:29.390993564 (26h13m15.174333663s) [upload=25083]: Waiting for error group managing segments...
May 26 04:17:29.673854379 (26h13m15.457194479s) [noise=true , address=212.102.52.215:4381 , piece_id=5UZ45SYYXIOMUYBHLRRP6NECZCQ5JW2RIK5MQIV2LMYQBVOON4ZA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.674402245 (26h13m15.457742344s) [noise=true , address=212.102.52.217:4433 , piece_id=FGN6M3NBSA3ZOLHYPT3THZV4THHNFYSI2R6BDB4AWSQZRCZ2M7TQ, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.677827470 (26h13m15.461167569s) [noise=true , address=212.102.52.213:4344 , piece_id=UEGMZ7I63LXBAKRWSFQDHBZBXPCHG2NMZSGW4PFVHFQORZLRMQYA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.678368041 (26h13m15.46170814s) [noise=true , address=212.102.52.215:4379 , piece_id=FBM4P5FPAP6OCVSHW5QYJOJWRXCO63HY2I6K3EYZKL6BRIUMME5Q, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.680215035 (26h13m15.463555134s) [noise=true , address=212.102.52.216:4402 , piece_id=VNTORDVWCQPSO5C2QUQUI76CBCTFB7TIQSEL7MWCCYRJ22T4WIBA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.682230351 (26h13m15.46557045s) [noise=true , address=212.102.52.217:4420 , piece_id=PQM3W3VAO5T2BBPQEREGSHGZ3EYBGJ7LFORGCKUY6NTUP2NNE5PA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.685255182 (26h13m15.468595281s) [noise=true , address=212.102.52.216:4405 , piece_id=65IGD4H7EMWEDYW2TFH643JJ5QW2BKHVFZA6D77MCHRYA6T6ESGQ, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.686472445 (26h13m15.469812544s) [noise=true , address=212.102.52.214:4362 , piece_id=LB7JGSWGNGU7BQX4FZQLVA2IUDQJT7ZP5PFXOQV4NLYXH7JAFRHA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.689410080 (26h13m15.472750179s) [noise=true , address=212.102.52.217:4436 , piece_id=SWQCVL47FBVYZFZIQVYNAD4NYDQLBBJL3S2EYVNHGS2Y54RRAN6A, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.689953157 (26h13m15.473293256s) [noise=true , address=212.102.52.214:4355 , piece_id=RKSUNVZS66X4A7ZY7WEUQXSWOTK2H5VAR7TBDICVC7MGVHA7LTGA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.694371046 (26h13m15.477711155s) [noise=true , address=212.102.52.214:4347 , piece_id=VPYOK5SVHEPZ36OOB7PYDAAINGIOUACK4FVDZBMJLWKMWC2D4NXQ, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.698718109 (26h13m15.482058208s) [noise=true , address=212.102.52.214:4349 , piece_id=HH47IVRCTHTDW4RHMJTEXGNB4ZK62NU6LDP5AS4FVXWPFINGHFNQ, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.699290191 (26h13m15.48263029s) [noise=true , address=212.102.52.215:4389 , piece_id=2OTWYJIV6A2646HAW5TWBTSQVT25V2PBKVANLGGK5SWNVHE3WGWA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.701256312 (26h13m15.484596411s) [noise=true , address=212.102.52.216:4397 , piece_id=5UNX4TPU63Q4AG726PSLVBFWX7ZZJL766ULEHEIIU3S4LG3EUGDA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.735255313 (26h13m15.518595412s) [noise=true , address=212.102.52.213:4331 , piece_id=LTU5YKIGVWFRQXGCECWGW656T6JMUDM7RRK6QD5CRGBB5IKMZXTQ, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.738998335 (26h13m15.522338434s) [noise=true , address=212.102.52.215:4380 , piece_id=6XJ45LCPGAIOCFHI35GODZRUNRS2Z63RF6WIKFUXV5R3GMSH7YJQ, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.739008105 (26h13m15.522348204s) [seg_pos={0 0}, upload=25083]: Satisfied cohort requirements after 16 pieces
May 26 04:17:29.739201553 (26h13m15.522541652s) [noise=true , address=212.102.52.217:4423 , piece_id=2KWQV3P3UYSFMBJ5BJNIHGVNZZXSCD3DEJXUV7SYVDALJSXIJAPA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TgGtX5XRdWH1qn6V5fZpQ8cY5oA2NyZx1AM17XANmr99AG9oh): protocol: expected piece hash; context canceled; EOF
May 26 04:17:29.739208446 (26h13m15.522548545s) [noise=true , address=212.102.52.217:4423 , piece_id=2KWQV3P3UYSFMBJ5BJNIHGVNZZXSCD3DEJXUV7SYVDALJSXIJAPA, seg_pos={0 0}, upload=25083]: Long tail cancellation due to optimal threshold
May 26 04:17:29.739277749 (26h13m15.522617858s) [noise=true , address=212.102.52.213:4343 , piece_id=AUT3LEUM7MPZIL33QGT47V7QOOZTHR3A2W2S6EMKJBY4B3AUA4ZA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129SbAN2aTSXzo8JLELV2ssdkmT5k7kbTcvxSZE4sFsVaKw1Aok): protocol: expected piece hash; context canceled; EOF
May 26 04:17:29.739288459 (26h13m15.522628598s) [noise=true , address=212.102.52.213:4343 , piece_id=AUT3LEUM7MPZIL33QGT47V7QOOZTHR3A2W2S6EMKJBY4B3AUA4ZA, seg_pos={0 0}, upload=25083]: Long tail cancellation due to optimal threshold
May 26 04:17:29.739324538 (26h13m15.522664637s) [noise=true , address=212.102.52.216:4408 , piece_id=23VTNWQ4FKNZ4O552CZQERQ5KMMHSCP4Y46PNSMWU4HGHSDFLIOA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BsdeZy1odVCAgcjxU6KdveXhUUTJKviRrrHLNCUuVergW6zVk): protocol: expected piece hash; context canceled; EOF
May 26 04:17:29.739330078 (26h13m15.522670177s) [noise=true , address=212.102.52.216:4408 , piece_id=23VTNWQ4FKNZ4O552CZQERQ5KMMHSCP4Y46PNSMWU4HGHSDFLIOA, seg_pos={0 0}, upload=25083]: Long tail cancellation due to optimal threshold
May 26 04:17:29.739420791 (26h13m15.52276089s) [noise=true , address=212.102.52.213:4333 , piece_id=O4MDU2BWFAOHKWNQ3KS24KNT6VYDQ4MGLBGCWW4WZZTPMVBT4OCQ, seg_pos={0 0}, upload=25083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Gm3eZxZPxPeHqLQ32kpXMpCgnwRoP8epBYtiFF1EH3U9LEazY): protocol: expected piece hash; context canceled; EOF
May 26 04:17:29.739426021 (26h13m15.52276612s) [noise=true , address=212.102.52.213:4333 , piece_id=O4MDU2BWFAOHKWNQ3KS24KNT6VYDQ4MGLBGCWW4WZZTPMVBT4OCQ, seg_pos={0 0}, upload=25083]: Long tail cancellation due to optimal threshold
May 26 04:17:29.739475536 (26h13m15.522815635s) [seg_pos={0 0}, upload=25083]: Done waiting for segment. successful: 16 optimal: 16 errs: <nil>
May 26 04:17:29.739535340 (26h13m15.522875439s) [upload=25083]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 26 04:17:29.739552654 (26h13m15.522892753s) [upload=25083]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 26 04:17:39.024892360 (26h13m24.808232459s) [upload=25083]: Done uploading segments...
May 26 04:17:39.024959289 (26h13m24.808299388s) [upload=25083]: Upload finished. err: metaclient: context deadline exceeded
Duration: 4.347134925s
core
uplink
uk-select-1
uk1-placement-17
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [............X.....................................X.........] duration:
3.642849042s trace:
https://jaeger.dev.storj.io/trace/2f0b656030b9f519
FAIL (2026-05-26 04:29:16 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.642866175s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)antiblip:
4 out of 60 is failed (threshold=3) [..X............X..........X................X................] duration:
4.607470855s trace:
https://jaeger.dev.storj.io/trace/20840979a8f7088f
FAIL (2026-05-26 04:28:39 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
25083]: Uploading piece...
May 26 04:17:28.318973560 (26h13m14.10231366s) [noise=true , address=212.102.52.217:4420 , piece_id=PQM3W3VAO5T2BBPQEREGSHGZ3EYBGJ7LFORGCKUY6NTUP2NNE5PA, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.319026442 (26h13m14.102366541s) [noise=true , address=212.102.52.216:4397 , piece_id=5UNX4TPU63Q4AG726PSLVBFWX7ZZJL766ULEHEIIU3S4LG3EUGDA, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.319245079 (26h13m14.102585178s) [noise=true , address=212.102.52.213:4343 , piece_id=AUT3LEUM7MPZIL33QGT47V7QOOZTHR3A2W2S6EMKJBY4B3AUA4ZA, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.319586399 (26h13m14.102926499s) [noise=true , address=212.102.52.216:4402 , piece_id=VNTORDVWCQPSO5C2QUQUI76CBCTFB7TIQSEL7MWCCYRJ22T4WIBA, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.319879910 (26h13m14.103220009s) [noise=true , address=212.102.52.217:4436 , piece_id=SWQCVL47FBVYZFZIQVYNAD4NYDQLBBJL3S2EYVNHGS2Y54RRAN6A, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.320133124 (26h13m14.103473223s) [noise=true , address=212.102.52.215:4379 , piece_id=FBM4P5FPAP6OCVSHW5QYJOJWRXCO63HY2I6K3EYZKL6BRIUMME5Q, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.320357862 (26h13m14.103697961s) [noise=true , address=212.102.52.214:4362 , piece_id=LB7JGSWGNGU7BQX4FZQLVA2IUDQJT7ZP5PFXOQV4NLYXH7JAFRHA, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.320571960 (26h13m14.103912059s) [noise=true , address=212.102.52.216:4408 , piece_id=23VTNWQ4FKNZ4O552CZQERQ5KMMHSCP4Y46PNSMWU4HGHSDFLIOA, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.320588152 (26h13m14.103928251s) [noise=true , address=212.102.52.213:4344 , piece_id=UEGMZ7I63LXBAKRWSFQDHBZBXPCHG2NMZSGW4PFVHFQORZLRMQYA, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.321070803 (26h13m14.104410912s) [noise=true , address=212.102.52.217:4433 , piece_id=FGN6M3NBSA3ZOLHYPT3THZV4THHNFYSI2R6BDB4AWSQZRCZ2M7TQ, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.321436691 (26h13m14.10477679s) [noise=true , address=212.102.52.215:4380 , piece_id=6XJ45LCPGAIOCFHI35GODZRUNRS2Z63RF6WIKFUXV5R3GMSH7YJQ, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.321852404 (26h13m14.105192503s) [noise=true , address=212.102.52.213:4333 , piece_id=O4MDU2BWFAOHKWNQ3KS24KNT6VYDQ4MGLBGCWW4WZZTPMVBT4OCQ, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.322253419 (26h13m14.105593518s) [noise=true , address=212.102.52.214:4347 , piece_id=VPYOK5SVHEPZ36OOB7PYDAAINGIOUACK4FVDZBMJLWKMWC2D4NXQ, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.322652791 (26h13m14.1059929s) [noise=true , address=212.102.52.213:4331 , piece_id=LTU5YKIGVWFRQXGCECWGW656T6JMUDM7RRK6QD5CRGBB5IKMZXTQ, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.324215282 (26h13m14.107555381s) [noise=true , address=212.102.52.214:4349 , piece_id=HH47IVRCTHTDW4RHMJTEXGNB4ZK62NU6LDP5AS4FVXWPFINGHFNQ, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:29.390980339 (26h13m15.174320438s) [upload=25083]: Next returned nil segment
May 26 04:17:29.390993564 (26h13m15.174333663s) [upload=25083]: Waiting for error group managing segments...
May 26 04:17:29.673854379 (26h13m15.457194479s) [noise=true , address=212.102.52.215:4381 , piece_id=5UZ45SYYXIOMUYBHLRRP6NECZCQ5JW2RIK5MQIV2LMYQBVOON4ZA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.674402245 (26h13m15.457742344s) [noise=true , address=212.102.52.217:4433 , piece_id=FGN6M3NBSA3ZOLHYPT3THZV4THHNFYSI2R6BDB4AWSQZRCZ2M7TQ, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.677827470 (26h13m15.461167569s) [noise=true , address=212.102.52.213:4344 , piece_id=UEGMZ7I63LXBAKRWSFQDHBZBXPCHG2NMZSGW4PFVHFQORZLRMQYA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.678368041 (26h13m15.46170814s) [noise=true , address=212.102.52.215:4379 , piece_id=FBM4P5FPAP6OCVSHW5QYJOJWRXCO63HY2I6K3EYZKL6BRIUMME5Q, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.680215035 (26h13m15.463555134s) [noise=true , address=212.102.52.216:4402 , piece_id=VNTORDVWCQPSO5C2QUQUI76CBCTFB7TIQSEL7MWCCYRJ22T4WIBA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.682230351 (26h13m15.46557045s) [noise=true , address=212.102.52.217:4420 , piece_id=PQM3W3VAO5T2BBPQEREGSHGZ3EYBGJ7LFORGCKUY6NTUP2NNE5PA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.685255182 (26h13m15.468595281s) [noise=true , address=212.102.52.216:4405 , piece_id=65IGD4H7EMWEDYW2TFH643JJ5QW2BKHVFZA6D77MCHRYA6T6ESGQ, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.686472445 (26h13m15.469812544s) [noise=true , address=212.102.52.214:4362 , piece_id=LB7JGSWGNGU7BQX4FZQLVA2IUDQJT7ZP5PFXOQV4NLYXH7JAFRHA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.689410080 (26h13m15.472750179s) [noise=true , address=212.102.52.217:4436 , piece_id=SWQCVL47FBVYZFZIQVYNAD4NYDQLBBJL3S2EYVNHGS2Y54RRAN6A, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.689953157 (26h13m15.473293256s) [noise=true , address=212.102.52.214:4355 , piece_id=RKSUNVZS66X4A7ZY7WEUQXSWOTK2H5VAR7TBDICVC7MGVHA7LTGA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.694371046 (26h13m15.477711155s) [noise=true , address=212.102.52.214:4347 , piece_id=VPYOK5SVHEPZ36OOB7PYDAAINGIOUACK4FVDZBMJLWKMWC2D4NXQ, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.698718109 (26h13m15.482058208s) [noise=true , address=212.102.52.214:4349 , piece_id=HH47IVRCTHTDW4RHMJTEXGNB4ZK62NU6LDP5AS4FVXWPFINGHFNQ, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.699290191 (26h13m15.48263029s) [noise=true , address=212.102.52.215:4389 , piece_id=2OTWYJIV6A2646HAW5TWBTSQVT25V2PBKVANLGGK5SWNVHE3WGWA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.701256312 (26h13m15.484596411s) [noise=true , address=212.102.52.216:4397 , piece_id=5UNX4TPU63Q4AG726PSLVBFWX7ZZJL766ULEHEIIU3S4LG3EUGDA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.735255313 (26h13m15.518595412s) [noise=true , address=212.102.52.213:4331 , piece_id=LTU5YKIGVWFRQXGCECWGW656T6JMUDM7RRK6QD5CRGBB5IKMZXTQ, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.738998335 (26h13m15.522338434s) [noise=true , address=212.102.52.215:4380 , piece_id=6XJ45LCPGAIOCFHI35GODZRUNRS2Z63RF6WIKFUXV5R3GMSH7YJQ, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.739008105 (26h13m15.522348204s) [seg_pos={0 0}, upload=25083]: Satisfied cohort requirements after 16 pieces
May 26 04:17:29.739201553 (26h13m15.522541652s) [noise=true , address=212.102.52.217:4423 , piece_id=2KWQV3P3UYSFMBJ5BJNIHGVNZZXSCD3DEJXUV7SYVDALJSXIJAPA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TgGtX5XRdWH1qn6V5fZpQ8cY5oA2NyZx1AM17XANmr99AG9oh): protocol: expected piece hash; context canceled; EOF
May 26 04:17:29.739208446 (26h13m15.522548545s) [noise=true , address=212.102.52.217:4423 , piece_id=2KWQV3P3UYSFMBJ5BJNIHGVNZZXSCD3DEJXUV7SYVDALJSXIJAPA, seg_pos={0 0}, upload=25083]: Long tail cancellation due to optimal threshold
May 26 04:17:29.739277749 (26h13m15.522617858s) [noise=true , address=212.102.52.213:4343 , piece_id=AUT3LEUM7MPZIL33QGT47V7QOOZTHR3A2W2S6EMKJBY4B3AUA4ZA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129SbAN2aTSXzo8JLELV2ssdkmT5k7kbTcvxSZE4sFsVaKw1Aok): protocol: expected piece hash; context canceled; EOF
May 26 04:17:29.739288459 (26h13m15.522628598s) [noise=true , address=212.102.52.213:4343 , piece_id=AUT3LEUM7MPZIL33QGT47V7QOOZTHR3A2W2S6EMKJBY4B3AUA4ZA, seg_pos={0 0}, upload=25083]: Long tail cancellation due to optimal threshold
May 26 04:17:29.739324538 (26h13m15.522664637s) [noise=true , address=212.102.52.216:4408 , piece_id=23VTNWQ4FKNZ4O552CZQERQ5KMMHSCP4Y46PNSMWU4HGHSDFLIOA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BsdeZy1odVCAgcjxU6KdveXhUUTJKviRrrHLNCUuVergW6zVk): protocol: expected piece hash; context canceled; EOF
May 26 04:17:29.739330078 (26h13m15.522670177s) [noise=true , address=212.102.52.216:4408 , piece_id=23VTNWQ4FKNZ4O552CZQERQ5KMMHSCP4Y46PNSMWU4HGHSDFLIOA, seg_pos={0 0}, upload=25083]: Long tail cancellation due to optimal threshold
May 26 04:17:29.739420791 (26h13m15.52276089s) [noise=true , address=212.102.52.213:4333 , piece_id=O4MDU2BWFAOHKWNQ3KS24KNT6VYDQ4MGLBGCWW4WZZTPMVBT4OCQ, seg_pos={0 0}, upload=25083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Gm3eZxZPxPeHqLQ32kpXMpCgnwRoP8epBYtiFF1EH3U9LEazY): protocol: expected piece hash; context canceled; EOF
May 26 04:17:29.739426021 (26h13m15.52276612s) [noise=true , address=212.102.52.213:4333 , piece_id=O4MDU2BWFAOHKWNQ3KS24KNT6VYDQ4MGLBGCWW4WZZTPMVBT4OCQ, seg_pos={0 0}, upload=25083]: Long tail cancellation due to optimal threshold
May 26 04:17:29.739475536 (26h13m15.522815635s) [seg_pos={0 0}, upload=25083]: Done waiting for segment. successful: 16 optimal: 16 errs: <nil>
May 26 04:17:29.739535340 (26h13m15.522875439s) [upload=25083]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 26 04:17:29.739552654 (26h13m15.522892753s) [upload=25083]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 26 04:17:39.024892360 (26h13m24.808232459s) [upload=25083]: Done uploading segments...
May 26 04:17:39.024959289 (26h13m24.808299388s) [upload=25083]: Upload finished. err: metaclient: context deadline exceeded
Duration: 4.607493268s
core
uplink
uk-select-1
uk1-placement-17
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [.............X.....................................X........] duration:
3.717755815s trace:
https://jaeger.dev.storj.io/trace/2f0b656030b9f519
FAIL (2026-05-26 04:28:33 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.717775361s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [..............X.....................................X.......] duration:
3.776960427s trace:
https://jaeger.dev.storj.io/trace/2f0b656030b9f519
FAIL (2026-05-26 04:28:03 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.776984283s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)antiblip:
4 out of 60 is failed (threshold=3) [...X............X..........X................X...............] duration:
4.002588288s trace:
https://jaeger.dev.storj.io/trace/20840979a8f7088f
FAIL (2026-05-26 04:27:57 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
25083]: Uploading piece...
May 26 04:17:28.318973560 (26h13m14.10231366s) [noise=true , address=212.102.52.217:4420 , piece_id=PQM3W3VAO5T2BBPQEREGSHGZ3EYBGJ7LFORGCKUY6NTUP2NNE5PA, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.319026442 (26h13m14.102366541s) [noise=true , address=212.102.52.216:4397 , piece_id=5UNX4TPU63Q4AG726PSLVBFWX7ZZJL766ULEHEIIU3S4LG3EUGDA, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.319245079 (26h13m14.102585178s) [noise=true , address=212.102.52.213:4343 , piece_id=AUT3LEUM7MPZIL33QGT47V7QOOZTHR3A2W2S6EMKJBY4B3AUA4ZA, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.319586399 (26h13m14.102926499s) [noise=true , address=212.102.52.216:4402 , piece_id=VNTORDVWCQPSO5C2QUQUI76CBCTFB7TIQSEL7MWCCYRJ22T4WIBA, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.319879910 (26h13m14.103220009s) [noise=true , address=212.102.52.217:4436 , piece_id=SWQCVL47FBVYZFZIQVYNAD4NYDQLBBJL3S2EYVNHGS2Y54RRAN6A, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.320133124 (26h13m14.103473223s) [noise=true , address=212.102.52.215:4379 , piece_id=FBM4P5FPAP6OCVSHW5QYJOJWRXCO63HY2I6K3EYZKL6BRIUMME5Q, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.320357862 (26h13m14.103697961s) [noise=true , address=212.102.52.214:4362 , piece_id=LB7JGSWGNGU7BQX4FZQLVA2IUDQJT7ZP5PFXOQV4NLYXH7JAFRHA, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.320571960 (26h13m14.103912059s) [noise=true , address=212.102.52.216:4408 , piece_id=23VTNWQ4FKNZ4O552CZQERQ5KMMHSCP4Y46PNSMWU4HGHSDFLIOA, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.320588152 (26h13m14.103928251s) [noise=true , address=212.102.52.213:4344 , piece_id=UEGMZ7I63LXBAKRWSFQDHBZBXPCHG2NMZSGW4PFVHFQORZLRMQYA, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.321070803 (26h13m14.104410912s) [noise=true , address=212.102.52.217:4433 , piece_id=FGN6M3NBSA3ZOLHYPT3THZV4THHNFYSI2R6BDB4AWSQZRCZ2M7TQ, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.321436691 (26h13m14.10477679s) [noise=true , address=212.102.52.215:4380 , piece_id=6XJ45LCPGAIOCFHI35GODZRUNRS2Z63RF6WIKFUXV5R3GMSH7YJQ, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.321852404 (26h13m14.105192503s) [noise=true , address=212.102.52.213:4333 , piece_id=O4MDU2BWFAOHKWNQ3KS24KNT6VYDQ4MGLBGCWW4WZZTPMVBT4OCQ, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.322253419 (26h13m14.105593518s) [noise=true , address=212.102.52.214:4347 , piece_id=VPYOK5SVHEPZ36OOB7PYDAAINGIOUACK4FVDZBMJLWKMWC2D4NXQ, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.322652791 (26h13m14.1059929s) [noise=true , address=212.102.52.213:4331 , piece_id=LTU5YKIGVWFRQXGCECWGW656T6JMUDM7RRK6QD5CRGBB5IKMZXTQ, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.324215282 (26h13m14.107555381s) [noise=true , address=212.102.52.214:4349 , piece_id=HH47IVRCTHTDW4RHMJTEXGNB4ZK62NU6LDP5AS4FVXWPFINGHFNQ, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:29.390980339 (26h13m15.174320438s) [upload=25083]: Next returned nil segment
May 26 04:17:29.390993564 (26h13m15.174333663s) [upload=25083]: Waiting for error group managing segments...
May 26 04:17:29.673854379 (26h13m15.457194479s) [noise=true , address=212.102.52.215:4381 , piece_id=5UZ45SYYXIOMUYBHLRRP6NECZCQ5JW2RIK5MQIV2LMYQBVOON4ZA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.674402245 (26h13m15.457742344s) [noise=true , address=212.102.52.217:4433 , piece_id=FGN6M3NBSA3ZOLHYPT3THZV4THHNFYSI2R6BDB4AWSQZRCZ2M7TQ, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.677827470 (26h13m15.461167569s) [noise=true , address=212.102.52.213:4344 , piece_id=UEGMZ7I63LXBAKRWSFQDHBZBXPCHG2NMZSGW4PFVHFQORZLRMQYA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.678368041 (26h13m15.46170814s) [noise=true , address=212.102.52.215:4379 , piece_id=FBM4P5FPAP6OCVSHW5QYJOJWRXCO63HY2I6K3EYZKL6BRIUMME5Q, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.680215035 (26h13m15.463555134s) [noise=true , address=212.102.52.216:4402 , piece_id=VNTORDVWCQPSO5C2QUQUI76CBCTFB7TIQSEL7MWCCYRJ22T4WIBA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.682230351 (26h13m15.46557045s) [noise=true , address=212.102.52.217:4420 , piece_id=PQM3W3VAO5T2BBPQEREGSHGZ3EYBGJ7LFORGCKUY6NTUP2NNE5PA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.685255182 (26h13m15.468595281s) [noise=true , address=212.102.52.216:4405 , piece_id=65IGD4H7EMWEDYW2TFH643JJ5QW2BKHVFZA6D77MCHRYA6T6ESGQ, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.686472445 (26h13m15.469812544s) [noise=true , address=212.102.52.214:4362 , piece_id=LB7JGSWGNGU7BQX4FZQLVA2IUDQJT7ZP5PFXOQV4NLYXH7JAFRHA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.689410080 (26h13m15.472750179s) [noise=true , address=212.102.52.217:4436 , piece_id=SWQCVL47FBVYZFZIQVYNAD4NYDQLBBJL3S2EYVNHGS2Y54RRAN6A, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.689953157 (26h13m15.473293256s) [noise=true , address=212.102.52.214:4355 , piece_id=RKSUNVZS66X4A7ZY7WEUQXSWOTK2H5VAR7TBDICVC7MGVHA7LTGA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.694371046 (26h13m15.477711155s) [noise=true , address=212.102.52.214:4347 , piece_id=VPYOK5SVHEPZ36OOB7PYDAAINGIOUACK4FVDZBMJLWKMWC2D4NXQ, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.698718109 (26h13m15.482058208s) [noise=true , address=212.102.52.214:4349 , piece_id=HH47IVRCTHTDW4RHMJTEXGNB4ZK62NU6LDP5AS4FVXWPFINGHFNQ, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.699290191 (26h13m15.48263029s) [noise=true , address=212.102.52.215:4389 , piece_id=2OTWYJIV6A2646HAW5TWBTSQVT25V2PBKVANLGGK5SWNVHE3WGWA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.701256312 (26h13m15.484596411s) [noise=true , address=212.102.52.216:4397 , piece_id=5UNX4TPU63Q4AG726PSLVBFWX7ZZJL766ULEHEIIU3S4LG3EUGDA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.735255313 (26h13m15.518595412s) [noise=true , address=212.102.52.213:4331 , piece_id=LTU5YKIGVWFRQXGCECWGW656T6JMUDM7RRK6QD5CRGBB5IKMZXTQ, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.738998335 (26h13m15.522338434s) [noise=true , address=212.102.52.215:4380 , piece_id=6XJ45LCPGAIOCFHI35GODZRUNRS2Z63RF6WIKFUXV5R3GMSH7YJQ, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.739008105 (26h13m15.522348204s) [seg_pos={0 0}, upload=25083]: Satisfied cohort requirements after 16 pieces
May 26 04:17:29.739201553 (26h13m15.522541652s) [noise=true , address=212.102.52.217:4423 , piece_id=2KWQV3P3UYSFMBJ5BJNIHGVNZZXSCD3DEJXUV7SYVDALJSXIJAPA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TgGtX5XRdWH1qn6V5fZpQ8cY5oA2NyZx1AM17XANmr99AG9oh): protocol: expected piece hash; context canceled; EOF
May 26 04:17:29.739208446 (26h13m15.522548545s) [noise=true , address=212.102.52.217:4423 , piece_id=2KWQV3P3UYSFMBJ5BJNIHGVNZZXSCD3DEJXUV7SYVDALJSXIJAPA, seg_pos={0 0}, upload=25083]: Long tail cancellation due to optimal threshold
May 26 04:17:29.739277749 (26h13m15.522617858s) [noise=true , address=212.102.52.213:4343 , piece_id=AUT3LEUM7MPZIL33QGT47V7QOOZTHR3A2W2S6EMKJBY4B3AUA4ZA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129SbAN2aTSXzo8JLELV2ssdkmT5k7kbTcvxSZE4sFsVaKw1Aok): protocol: expected piece hash; context canceled; EOF
May 26 04:17:29.739288459 (26h13m15.522628598s) [noise=true , address=212.102.52.213:4343 , piece_id=AUT3LEUM7MPZIL33QGT47V7QOOZTHR3A2W2S6EMKJBY4B3AUA4ZA, seg_pos={0 0}, upload=25083]: Long tail cancellation due to optimal threshold
May 26 04:17:29.739324538 (26h13m15.522664637s) [noise=true , address=212.102.52.216:4408 , piece_id=23VTNWQ4FKNZ4O552CZQERQ5KMMHSCP4Y46PNSMWU4HGHSDFLIOA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BsdeZy1odVCAgcjxU6KdveXhUUTJKviRrrHLNCUuVergW6zVk): protocol: expected piece hash; context canceled; EOF
May 26 04:17:29.739330078 (26h13m15.522670177s) [noise=true , address=212.102.52.216:4408 , piece_id=23VTNWQ4FKNZ4O552CZQERQ5KMMHSCP4Y46PNSMWU4HGHSDFLIOA, seg_pos={0 0}, upload=25083]: Long tail cancellation due to optimal threshold
May 26 04:17:29.739420791 (26h13m15.52276089s) [noise=true , address=212.102.52.213:4333 , piece_id=O4MDU2BWFAOHKWNQ3KS24KNT6VYDQ4MGLBGCWW4WZZTPMVBT4OCQ, seg_pos={0 0}, upload=25083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Gm3eZxZPxPeHqLQ32kpXMpCgnwRoP8epBYtiFF1EH3U9LEazY): protocol: expected piece hash; context canceled; EOF
May 26 04:17:29.739426021 (26h13m15.52276612s) [noise=true , address=212.102.52.213:4333 , piece_id=O4MDU2BWFAOHKWNQ3KS24KNT6VYDQ4MGLBGCWW4WZZTPMVBT4OCQ, seg_pos={0 0}, upload=25083]: Long tail cancellation due to optimal threshold
May 26 04:17:29.739475536 (26h13m15.522815635s) [seg_pos={0 0}, upload=25083]: Done waiting for segment. successful: 16 optimal: 16 errs: <nil>
May 26 04:17:29.739535340 (26h13m15.522875439s) [upload=25083]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 26 04:17:29.739552654 (26h13m15.522892753s) [upload=25083]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 26 04:17:39.024892360 (26h13m24.808232459s) [upload=25083]: Done uploading segments...
May 26 04:17:39.024959289 (26h13m24.808299388s) [upload=25083]: Upload finished. err: metaclient: context deadline exceeded
Duration: 4.002608507s
core
uplink
uk-select-1
uk1-placement-17
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [...............X.....................................X......] duration:
4.057458013s trace:
https://jaeger.dev.storj.io/trace/2f0b656030b9f519
FAIL (2026-05-26 04:27:14 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 4.057581979s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)antiblip:
4 out of 60 is failed (threshold=3) [....X............X..........X................X..............] duration:
4.481174725s trace:
https://jaeger.dev.storj.io/trace/20840979a8f7088f
FAIL (2026-05-26 04:26:57 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
25083]: Uploading piece...
May 26 04:17:28.318973560 (26h13m14.10231366s) [noise=true , address=212.102.52.217:4420 , piece_id=PQM3W3VAO5T2BBPQEREGSHGZ3EYBGJ7LFORGCKUY6NTUP2NNE5PA, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.319026442 (26h13m14.102366541s) [noise=true , address=212.102.52.216:4397 , piece_id=5UNX4TPU63Q4AG726PSLVBFWX7ZZJL766ULEHEIIU3S4LG3EUGDA, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.319245079 (26h13m14.102585178s) [noise=true , address=212.102.52.213:4343 , piece_id=AUT3LEUM7MPZIL33QGT47V7QOOZTHR3A2W2S6EMKJBY4B3AUA4ZA, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.319586399 (26h13m14.102926499s) [noise=true , address=212.102.52.216:4402 , piece_id=VNTORDVWCQPSO5C2QUQUI76CBCTFB7TIQSEL7MWCCYRJ22T4WIBA, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.319879910 (26h13m14.103220009s) [noise=true , address=212.102.52.217:4436 , piece_id=SWQCVL47FBVYZFZIQVYNAD4NYDQLBBJL3S2EYVNHGS2Y54RRAN6A, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.320133124 (26h13m14.103473223s) [noise=true , address=212.102.52.215:4379 , piece_id=FBM4P5FPAP6OCVSHW5QYJOJWRXCO63HY2I6K3EYZKL6BRIUMME5Q, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.320357862 (26h13m14.103697961s) [noise=true , address=212.102.52.214:4362 , piece_id=LB7JGSWGNGU7BQX4FZQLVA2IUDQJT7ZP5PFXOQV4NLYXH7JAFRHA, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.320571960 (26h13m14.103912059s) [noise=true , address=212.102.52.216:4408 , piece_id=23VTNWQ4FKNZ4O552CZQERQ5KMMHSCP4Y46PNSMWU4HGHSDFLIOA, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.320588152 (26h13m14.103928251s) [noise=true , address=212.102.52.213:4344 , piece_id=UEGMZ7I63LXBAKRWSFQDHBZBXPCHG2NMZSGW4PFVHFQORZLRMQYA, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.321070803 (26h13m14.104410912s) [noise=true , address=212.102.52.217:4433 , piece_id=FGN6M3NBSA3ZOLHYPT3THZV4THHNFYSI2R6BDB4AWSQZRCZ2M7TQ, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.321436691 (26h13m14.10477679s) [noise=true , address=212.102.52.215:4380 , piece_id=6XJ45LCPGAIOCFHI35GODZRUNRS2Z63RF6WIKFUXV5R3GMSH7YJQ, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.321852404 (26h13m14.105192503s) [noise=true , address=212.102.52.213:4333 , piece_id=O4MDU2BWFAOHKWNQ3KS24KNT6VYDQ4MGLBGCWW4WZZTPMVBT4OCQ, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.322253419 (26h13m14.105593518s) [noise=true , address=212.102.52.214:4347 , piece_id=VPYOK5SVHEPZ36OOB7PYDAAINGIOUACK4FVDZBMJLWKMWC2D4NXQ, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.322652791 (26h13m14.1059929s) [noise=true , address=212.102.52.213:4331 , piece_id=LTU5YKIGVWFRQXGCECWGW656T6JMUDM7RRK6QD5CRGBB5IKMZXTQ, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.324215282 (26h13m14.107555381s) [noise=true , address=212.102.52.214:4349 , piece_id=HH47IVRCTHTDW4RHMJTEXGNB4ZK62NU6LDP5AS4FVXWPFINGHFNQ, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:29.390980339 (26h13m15.174320438s) [upload=25083]: Next returned nil segment
May 26 04:17:29.390993564 (26h13m15.174333663s) [upload=25083]: Waiting for error group managing segments...
May 26 04:17:29.673854379 (26h13m15.457194479s) [noise=true , address=212.102.52.215:4381 , piece_id=5UZ45SYYXIOMUYBHLRRP6NECZCQ5JW2RIK5MQIV2LMYQBVOON4ZA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.674402245 (26h13m15.457742344s) [noise=true , address=212.102.52.217:4433 , piece_id=FGN6M3NBSA3ZOLHYPT3THZV4THHNFYSI2R6BDB4AWSQZRCZ2M7TQ, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.677827470 (26h13m15.461167569s) [noise=true , address=212.102.52.213:4344 , piece_id=UEGMZ7I63LXBAKRWSFQDHBZBXPCHG2NMZSGW4PFVHFQORZLRMQYA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.678368041 (26h13m15.46170814s) [noise=true , address=212.102.52.215:4379 , piece_id=FBM4P5FPAP6OCVSHW5QYJOJWRXCO63HY2I6K3EYZKL6BRIUMME5Q, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.680215035 (26h13m15.463555134s) [noise=true , address=212.102.52.216:4402 , piece_id=VNTORDVWCQPSO5C2QUQUI76CBCTFB7TIQSEL7MWCCYRJ22T4WIBA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.682230351 (26h13m15.46557045s) [noise=true , address=212.102.52.217:4420 , piece_id=PQM3W3VAO5T2BBPQEREGSHGZ3EYBGJ7LFORGCKUY6NTUP2NNE5PA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.685255182 (26h13m15.468595281s) [noise=true , address=212.102.52.216:4405 , piece_id=65IGD4H7EMWEDYW2TFH643JJ5QW2BKHVFZA6D77MCHRYA6T6ESGQ, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.686472445 (26h13m15.469812544s) [noise=true , address=212.102.52.214:4362 , piece_id=LB7JGSWGNGU7BQX4FZQLVA2IUDQJT7ZP5PFXOQV4NLYXH7JAFRHA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.689410080 (26h13m15.472750179s) [noise=true , address=212.102.52.217:4436 , piece_id=SWQCVL47FBVYZFZIQVYNAD4NYDQLBBJL3S2EYVNHGS2Y54RRAN6A, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.689953157 (26h13m15.473293256s) [noise=true , address=212.102.52.214:4355 , piece_id=RKSUNVZS66X4A7ZY7WEUQXSWOTK2H5VAR7TBDICVC7MGVHA7LTGA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.694371046 (26h13m15.477711155s) [noise=true , address=212.102.52.214:4347 , piece_id=VPYOK5SVHEPZ36OOB7PYDAAINGIOUACK4FVDZBMJLWKMWC2D4NXQ, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.698718109 (26h13m15.482058208s) [noise=true , address=212.102.52.214:4349 , piece_id=HH47IVRCTHTDW4RHMJTEXGNB4ZK62NU6LDP5AS4FVXWPFINGHFNQ, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.699290191 (26h13m15.48263029s) [noise=true , address=212.102.52.215:4389 , piece_id=2OTWYJIV6A2646HAW5TWBTSQVT25V2PBKVANLGGK5SWNVHE3WGWA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.701256312 (26h13m15.484596411s) [noise=true , address=212.102.52.216:4397 , piece_id=5UNX4TPU63Q4AG726PSLVBFWX7ZZJL766ULEHEIIU3S4LG3EUGDA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.735255313 (26h13m15.518595412s) [noise=true , address=212.102.52.213:4331 , piece_id=LTU5YKIGVWFRQXGCECWGW656T6JMUDM7RRK6QD5CRGBB5IKMZXTQ, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.738998335 (26h13m15.522338434s) [noise=true , address=212.102.52.215:4380 , piece_id=6XJ45LCPGAIOCFHI35GODZRUNRS2Z63RF6WIKFUXV5R3GMSH7YJQ, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.739008105 (26h13m15.522348204s) [seg_pos={0 0}, upload=25083]: Satisfied cohort requirements after 16 pieces
May 26 04:17:29.739201553 (26h13m15.522541652s) [noise=true , address=212.102.52.217:4423 , piece_id=2KWQV3P3UYSFMBJ5BJNIHGVNZZXSCD3DEJXUV7SYVDALJSXIJAPA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TgGtX5XRdWH1qn6V5fZpQ8cY5oA2NyZx1AM17XANmr99AG9oh): protocol: expected piece hash; context canceled; EOF
May 26 04:17:29.739208446 (26h13m15.522548545s) [noise=true , address=212.102.52.217:4423 , piece_id=2KWQV3P3UYSFMBJ5BJNIHGVNZZXSCD3DEJXUV7SYVDALJSXIJAPA, seg_pos={0 0}, upload=25083]: Long tail cancellation due to optimal threshold
May 26 04:17:29.739277749 (26h13m15.522617858s) [noise=true , address=212.102.52.213:4343 , piece_id=AUT3LEUM7MPZIL33QGT47V7QOOZTHR3A2W2S6EMKJBY4B3AUA4ZA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129SbAN2aTSXzo8JLELV2ssdkmT5k7kbTcvxSZE4sFsVaKw1Aok): protocol: expected piece hash; context canceled; EOF
May 26 04:17:29.739288459 (26h13m15.522628598s) [noise=true , address=212.102.52.213:4343 , piece_id=AUT3LEUM7MPZIL33QGT47V7QOOZTHR3A2W2S6EMKJBY4B3AUA4ZA, seg_pos={0 0}, upload=25083]: Long tail cancellation due to optimal threshold
May 26 04:17:29.739324538 (26h13m15.522664637s) [noise=true , address=212.102.52.216:4408 , piece_id=23VTNWQ4FKNZ4O552CZQERQ5KMMHSCP4Y46PNSMWU4HGHSDFLIOA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BsdeZy1odVCAgcjxU6KdveXhUUTJKviRrrHLNCUuVergW6zVk): protocol: expected piece hash; context canceled; EOF
May 26 04:17:29.739330078 (26h13m15.522670177s) [noise=true , address=212.102.52.216:4408 , piece_id=23VTNWQ4FKNZ4O552CZQERQ5KMMHSCP4Y46PNSMWU4HGHSDFLIOA, seg_pos={0 0}, upload=25083]: Long tail cancellation due to optimal threshold
May 26 04:17:29.739420791 (26h13m15.52276089s) [noise=true , address=212.102.52.213:4333 , piece_id=O4MDU2BWFAOHKWNQ3KS24KNT6VYDQ4MGLBGCWW4WZZTPMVBT4OCQ, seg_pos={0 0}, upload=25083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Gm3eZxZPxPeHqLQ32kpXMpCgnwRoP8epBYtiFF1EH3U9LEazY): protocol: expected piece hash; context canceled; EOF
May 26 04:17:29.739426021 (26h13m15.52276612s) [noise=true , address=212.102.52.213:4333 , piece_id=O4MDU2BWFAOHKWNQ3KS24KNT6VYDQ4MGLBGCWW4WZZTPMVBT4OCQ, seg_pos={0 0}, upload=25083]: Long tail cancellation due to optimal threshold
May 26 04:17:29.739475536 (26h13m15.522815635s) [seg_pos={0 0}, upload=25083]: Done waiting for segment. successful: 16 optimal: 16 errs: <nil>
May 26 04:17:29.739535340 (26h13m15.522875439s) [upload=25083]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 26 04:17:29.739552654 (26h13m15.522892753s) [upload=25083]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 26 04:17:39.024892360 (26h13m24.808232459s) [upload=25083]: Done uploading segments...
May 26 04:17:39.024959289 (26h13m24.808299388s) [upload=25083]: Upload finished. err: metaclient: context deadline exceeded
Duration: 4.481195534s
core
uplink
uk-select-1
uk1-placement-17
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [................X.....................................X.....] duration:
3.840018713s trace:
https://jaeger.dev.storj.io/trace/2f0b656030b9f519
FAIL (2026-05-26 04:26:41 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.840081982s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [X...........................................................] duration:
7.241372339s trace:
https://jaeger.dev.storj.io/trace/2f99fbcd176b1617
FAIL (2026-05-26 04:26:28 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
May 26 03:37:44.554844381 (25h33m30.33818448s) [upload=24452]: Starting upload
May 26 03:37:44.554914775 (25h33m30.338254884s) [upload=24452]: Done starting upload
May 26 03:37:44.555010087 (25h33m30.338350186s) [upload=24452]: Uploading segments...
May 26 03:37:44.555154713 (25h33m30.338494812s) [upload=24452]: Got next segment. Inline: false
May 26 03:37:44.555190892 (25h33m30.338530991s) [upload=24452]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
May 26 03:37:44.555194178 (25h33m30.338534277s) [upload=24452]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
May 26 03:37:44.796393493 (25h33m30.579733593s) [seg_pos={0 0}, upload=24452]: Begin upload segment...
May 26 03:37:44.796434813 (25h33m30.579774902s) [seg_pos={0 0}, upload=24452]: Done begin upload segment.
May 26 03:37:44.796455061 (25h33m30.57979516s) [noise=true , address=145.79.227.26:4657 , piece_id=JCI2K722O5F2HGZR4POW3UMUCGPXQUHMSZFGJL7IL2TSY3T4LKNA, seg_pos={0 0}, upload=24452]: Uploading piece...
May 26 03:37:44.796479167 (25h33m30.579819266s) [noise=true , address=77.90.31.11:4558 , piece_id=SGKGDSAPYDNZ2KDMXJXCQTENMFPAGI6VAXZY3WQUBIZSRFZHPWZQ, seg_pos={0 0}, upload=24452]: Uploading piece...
May 26 03:37:44.796547607 (25h33m30.579887706s) [noise=true , address=145.79.227.15:4873 , piece_id=MCX4ZRVNDVT53WE7W5X7JZ53ZFVOJIUWMZBPHCXAZBUXS4RULCTA, seg_pos={0 0}, upload=24452]: Uploading piece...
May 26 03:37:44.796944846 (25h33m30.580284935s) [noise=true , address=145.79.227.5:4224 , piece_id=QYELKAAUIN72JYBUBRWGKGOF5D4FZ6M4CV2LIRWYMX6Q3FU2TECA, seg_pos={0 0}, upload=24452]: Uploading piece...
May 26 03:37:44.797283402 (25h33m30.580623501s) [noise=true , address=77.90.31.16:4735 , piece_id=EHMPUNQ3X4KQ3OS3Y2ZSAN6QLRGPS2W6DL635MO7IN6FO7QX7LFQ, seg_pos={0 0}, upload=24452]: Uploading piece...
May 26 03:37:44.797625474 (25h33m30.580965573s) [noise=true , address=145.79.227.40:4049 , piece_id=BAAQV3TB3GMDZNLG6UUEMPNJ2R5E5RVSEFBTOPDQHNUXJFB6XOLA, seg_pos={0 0}, upload=24452]: Uploading piece...
May 26 03:37:44.798009868 (25h33m30.581349967s) [noise=true , address=145.79.227.8:4332 , piece_id=LRJG7OY6SCUWAEATGPICV3SQLID25AAPL5DJHMRX4T66MOVT4ARQ, seg_pos={0 0}, upload=24452]: Uploading piece...
May 26 03:37:44.798332895 (25h33m30.581672994s) [noise=true , address=145.79.227.45:4234 , piece_id=2YBD3GONCSIP3A63LSVRI647YC3NTWSXUCBY5ZWEIWMT5B7D7YLQ, seg_pos={0 0}, upload=24452]: Uploading piece...
May 26 03:37:44.798639480 (25h33m30.581979569s) [noise=true , address=145.79.227.10:4401 , piece_id=FDK7SAUVXCQM2XHWJQMLHMLEVYVYH3FWGR4CI5KLFPPFZJRCNJKQ, seg_pos={0 0}, upload=24452]: Uploading piece...
May 26 03:37:44.799006550 (25h33m30.582346649s) [noise=true , address=45.91.116.12:4227 , piece_id=VUZQ6SGI6MGKMHZOT4KZP2Z2CTUGHTQ6373WQRZPTN2KANTEGDTA, seg_pos={0 0}, upload=24452]: Uploading piece...
May 26 03:37:44.799936012 (25h33m30.583276102s) [noise=true , address=45.91.116.4:4947 , piece_id=D2SJUTVUCPKJ2UL2UVZPCGITIXAIMYGDVSSD3Z5XUUN7SB6WAADA, seg_pos={0 0}, upload=24452]: Uploading piece...
May 26 03:37:44.801075256 (25h33m30.584415356s) [noise=true , address=145.79.227.12:4769 , piece_id=4CMAB7IQW3DY72X4R2M4D3GEDZXHQHWYEVBN3VKXMSSYABFOMJWA, seg_pos={0 0}, upload=24452]: Uploading piece...
May 26 03:37:44.802249427 (25h33m30.585589526s) [noise=true , address=145.79.227.21:4473 , piece_id=CV3E5SXKUA3OAW2QRQHHNDSK74MYME6AUNE7YD2YQJMP6MAVSQEQ, seg_pos={0 0}, upload=24452]: Uploading piece...
May 26 03:37:44.803442773 (25h33m30.586782872s) [noise=true , address=145.79.227.49:4340 , piece_id=H4PP2BCFBCRUPINAFKPEIS7IKRCDVILC4EPYEEWNTBSHTNICWGGA, seg_pos={0 0}, upload=24452]: Uploading piece...
May 26 03:37:44.804531642 (25h33m30.587871741s) [noise=true , address=77.90.31.17:4773 , piece_id=45PHIJTVNU5KTCWK4RUCJ3MF2TJSA4B37W4CG6UJ3GFRRPQQ5WZA, seg_pos={0 0}, upload=24452]: Uploading piece...
May 26 03:37:44.805927776 (25h33m30.589267875s) [noise=true , address=77.90.31.12:4589 , piece_id=P6TPHQWR7IOYUKZ4VZKC6CH77YAJJUU2A33WTECMQPUVI2ZCUQYA, seg_pos={0 0}, upload=24452]: Uploading piece...
May 26 03:37:46.520818854 (25h33m32.304158953s) [upload=24452]: Next returned nil segment
May 26 03:37:46.520830196 (25h33m32.304170295s) [upload=24452]: Waiting for error group managing segments...
May 26 03:37:46.802301060 (25h33m32.585641159s) [noise=true , address=145.79.227.8:4332 , piece_id=LRJG7OY6SCUWAEATGPICV3SQLID25AAPL5DJHMRX4T66MOVT4ARQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil>
May 26 03:37:46.805062139 (25h33m32.588402238s) [noise=true , address=145.79.227.12:4769 , piece_id=4CMAB7IQW3DY72X4R2M4D3GEDZXHQHWYEVBN3VKXMSSYABFOMJWA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil>
May 26 03:37:46.818069497 (25h33m32.601409586s) [noise=true , address=145.79.227.49:4340 , piece_id=H4PP2BCFBCRUPINAFKPEIS7IKRCDVILC4EPYEEWNTBSHTNICWGGA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil>
May 26 03:37:46.818763521 (25h33m32.60210362s) [noise=true , address=145.79.227.26:4657 , piece_id=JCI2K722O5F2HGZR4POW3UMUCGPXQUHMSZFGJL7IL2TSY3T4LKNA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil>
May 26 03:37:46.832663342 (25h33m32.616003451s) [noise=true , address=145.79.227.21:4473 , piece_id=CV3E5SXKUA3OAW2QRQHHNDSK74MYME6AUNE7YD2YQJMP6MAVSQEQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil>
May 26 03:37:46.841016390 (25h33m32.624356489s) [noise=true , address=145.79.227.45:4234 , piece_id=2YBD3GONCSIP3A63LSVRI647YC3NTWSXUCBY5ZWEIWMT5B7D7YLQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil>
May 26 03:37:46.842493007 (25h33m32.625833106s) [noise=true , address=145.79.227.40:4049 , piece_id=BAAQV3TB3GMDZNLG6UUEMPNJ2R5E5RVSEFBTOPDQHNUXJFB6XOLA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil>
May 26 03:37:46.855901301 (25h33m32.63924139s) [noise=true , address=145.79.227.5:4224 , piece_id=QYELKAAUIN72JYBUBRWGKGOF5D4FZ6M4CV2LIRWYMX6Q3FU2TECA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil>
May 26 03:37:46.872794464 (25h33m32.656134563s) [noise=true , address=145.79.227.10:4401 , piece_id=FDK7SAUVXCQM2XHWJQMLHMLEVYVYH3FWGR4CI5KLFPPFZJRCNJKQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil>
May 26 03:37:46.904359019 (25h33m32.687699148s) [noise=true , address=145.79.227.15:4873 , piece_id=MCX4ZRVNDVT53WE7W5X7JZ53ZFVOJIUWMZBPHCXAZBUXS4RULCTA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil>
May 26 03:37:47.599588789 (25h33m33.382928888s) [noise=true , address=45.91.116.12:4227 , piece_id=VUZQ6SGI6MGKMHZOT4KZP2Z2CTUGHTQ6373WQRZPTN2KANTEGDTA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil>
May 26 03:37:47.616657837 (25h33m33.399997936s) [noise=true , address=45.91.116.4:4947 , piece_id=D2SJUTVUCPKJ2UL2UVZPCGITIXAIMYGDVSSD3Z5XUUN7SB6WAADA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil>
May 26 03:38:13.890981361 (25h33m59.67432147s) [noise=true , address=77.90.31.12:4589 , piece_id=P6TPHQWR7IOYUKZ4VZKC6CH77YAJJUU2A33WTECMQPUVI2ZCUQYA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: rpc: dial tcp 77.90.31.12:4589: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
May 26 03:38:13.891042047 (25h33m59.674382146s) [noise=true , address=77.90.31.16:4735 , piece_id=EHMPUNQ3X4KQ3OS3Y2ZSAN6QLRGPS2W6DL635MO7IN6FO7QX7LFQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: rpc: dial tcp 77.90.31.16:4735: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
May 26 03:38:13.891153340 (25h33m59.674493439s) [noise=true , address=77.90.31.17:4773 , piece_id=45PHIJTVNU5KTCWK4RUCJ3MF2TJSA4B37W4CG6UJ3GFRRPQQ5WZA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: ecclient: upload failed (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, address:77.90.31.17:4773): protocol: context deadline exceeded; context deadline exceeded
May 26 03:38:13.891203274 (25h33m59.674543393s) [noise=true , address=77.90.31.11:4558 , piece_id=SGKGDSAPYDNZ2KDMXJXCQTENMFPAGI6VAXZY3WQUBIZSRFZHPWZQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: ecclient: upload failed (node:1mGEP1n1icC2gEkyRVy6BeM4vsrgqYWaQeB5rUtxjrWQjyFhTS, address:77.90.31.11:4558): protocol: write tcp 188.166.225.170:40140->77.90.31.11:4558: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
May 26 03:38:13.891231057 (25h33m59.674571156s) [seg_pos={0 0}, upload=24452]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
May 26 03:38:14.130774583 (25h33m59.914114683s) [upload=24452]: Done uploading segments...
May 26 03:38:14.130801014 (25h33m59.914141113s) [upload=24452]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 7.241395112s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)antiblip:
4 out of 60 is failed (threshold=3) [.....X............X..........X................X.............] duration:
14.753975143s trace:
https://jaeger.dev.storj.io/trace/20840979a8f7088f
FAIL (2026-05-26 04:25:58 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
25083]: Uploading piece...
May 26 04:17:28.318973560 (26h13m14.10231366s) [noise=true , address=212.102.52.217:4420 , piece_id=PQM3W3VAO5T2BBPQEREGSHGZ3EYBGJ7LFORGCKUY6NTUP2NNE5PA, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.319026442 (26h13m14.102366541s) [noise=true , address=212.102.52.216:4397 , piece_id=5UNX4TPU63Q4AG726PSLVBFWX7ZZJL766ULEHEIIU3S4LG3EUGDA, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.319245079 (26h13m14.102585178s) [noise=true , address=212.102.52.213:4343 , piece_id=AUT3LEUM7MPZIL33QGT47V7QOOZTHR3A2W2S6EMKJBY4B3AUA4ZA, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.319586399 (26h13m14.102926499s) [noise=true , address=212.102.52.216:4402 , piece_id=VNTORDVWCQPSO5C2QUQUI76CBCTFB7TIQSEL7MWCCYRJ22T4WIBA, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.319879910 (26h13m14.103220009s) [noise=true , address=212.102.52.217:4436 , piece_id=SWQCVL47FBVYZFZIQVYNAD4NYDQLBBJL3S2EYVNHGS2Y54RRAN6A, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.320133124 (26h13m14.103473223s) [noise=true , address=212.102.52.215:4379 , piece_id=FBM4P5FPAP6OCVSHW5QYJOJWRXCO63HY2I6K3EYZKL6BRIUMME5Q, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.320357862 (26h13m14.103697961s) [noise=true , address=212.102.52.214:4362 , piece_id=LB7JGSWGNGU7BQX4FZQLVA2IUDQJT7ZP5PFXOQV4NLYXH7JAFRHA, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.320571960 (26h13m14.103912059s) [noise=true , address=212.102.52.216:4408 , piece_id=23VTNWQ4FKNZ4O552CZQERQ5KMMHSCP4Y46PNSMWU4HGHSDFLIOA, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.320588152 (26h13m14.103928251s) [noise=true , address=212.102.52.213:4344 , piece_id=UEGMZ7I63LXBAKRWSFQDHBZBXPCHG2NMZSGW4PFVHFQORZLRMQYA, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.321070803 (26h13m14.104410912s) [noise=true , address=212.102.52.217:4433 , piece_id=FGN6M3NBSA3ZOLHYPT3THZV4THHNFYSI2R6BDB4AWSQZRCZ2M7TQ, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.321436691 (26h13m14.10477679s) [noise=true , address=212.102.52.215:4380 , piece_id=6XJ45LCPGAIOCFHI35GODZRUNRS2Z63RF6WIKFUXV5R3GMSH7YJQ, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.321852404 (26h13m14.105192503s) [noise=true , address=212.102.52.213:4333 , piece_id=O4MDU2BWFAOHKWNQ3KS24KNT6VYDQ4MGLBGCWW4WZZTPMVBT4OCQ, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.322253419 (26h13m14.105593518s) [noise=true , address=212.102.52.214:4347 , piece_id=VPYOK5SVHEPZ36OOB7PYDAAINGIOUACK4FVDZBMJLWKMWC2D4NXQ, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.322652791 (26h13m14.1059929s) [noise=true , address=212.102.52.213:4331 , piece_id=LTU5YKIGVWFRQXGCECWGW656T6JMUDM7RRK6QD5CRGBB5IKMZXTQ, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.324215282 (26h13m14.107555381s) [noise=true , address=212.102.52.214:4349 , piece_id=HH47IVRCTHTDW4RHMJTEXGNB4ZK62NU6LDP5AS4FVXWPFINGHFNQ, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:29.390980339 (26h13m15.174320438s) [upload=25083]: Next returned nil segment
May 26 04:17:29.390993564 (26h13m15.174333663s) [upload=25083]: Waiting for error group managing segments...
May 26 04:17:29.673854379 (26h13m15.457194479s) [noise=true , address=212.102.52.215:4381 , piece_id=5UZ45SYYXIOMUYBHLRRP6NECZCQ5JW2RIK5MQIV2LMYQBVOON4ZA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.674402245 (26h13m15.457742344s) [noise=true , address=212.102.52.217:4433 , piece_id=FGN6M3NBSA3ZOLHYPT3THZV4THHNFYSI2R6BDB4AWSQZRCZ2M7TQ, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.677827470 (26h13m15.461167569s) [noise=true , address=212.102.52.213:4344 , piece_id=UEGMZ7I63LXBAKRWSFQDHBZBXPCHG2NMZSGW4PFVHFQORZLRMQYA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.678368041 (26h13m15.46170814s) [noise=true , address=212.102.52.215:4379 , piece_id=FBM4P5FPAP6OCVSHW5QYJOJWRXCO63HY2I6K3EYZKL6BRIUMME5Q, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.680215035 (26h13m15.463555134s) [noise=true , address=212.102.52.216:4402 , piece_id=VNTORDVWCQPSO5C2QUQUI76CBCTFB7TIQSEL7MWCCYRJ22T4WIBA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.682230351 (26h13m15.46557045s) [noise=true , address=212.102.52.217:4420 , piece_id=PQM3W3VAO5T2BBPQEREGSHGZ3EYBGJ7LFORGCKUY6NTUP2NNE5PA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.685255182 (26h13m15.468595281s) [noise=true , address=212.102.52.216:4405 , piece_id=65IGD4H7EMWEDYW2TFH643JJ5QW2BKHVFZA6D77MCHRYA6T6ESGQ, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.686472445 (26h13m15.469812544s) [noise=true , address=212.102.52.214:4362 , piece_id=LB7JGSWGNGU7BQX4FZQLVA2IUDQJT7ZP5PFXOQV4NLYXH7JAFRHA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.689410080 (26h13m15.472750179s) [noise=true , address=212.102.52.217:4436 , piece_id=SWQCVL47FBVYZFZIQVYNAD4NYDQLBBJL3S2EYVNHGS2Y54RRAN6A, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.689953157 (26h13m15.473293256s) [noise=true , address=212.102.52.214:4355 , piece_id=RKSUNVZS66X4A7ZY7WEUQXSWOTK2H5VAR7TBDICVC7MGVHA7LTGA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.694371046 (26h13m15.477711155s) [noise=true , address=212.102.52.214:4347 , piece_id=VPYOK5SVHEPZ36OOB7PYDAAINGIOUACK4FVDZBMJLWKMWC2D4NXQ, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.698718109 (26h13m15.482058208s) [noise=true , address=212.102.52.214:4349 , piece_id=HH47IVRCTHTDW4RHMJTEXGNB4ZK62NU6LDP5AS4FVXWPFINGHFNQ, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.699290191 (26h13m15.48263029s) [noise=true , address=212.102.52.215:4389 , piece_id=2OTWYJIV6A2646HAW5TWBTSQVT25V2PBKVANLGGK5SWNVHE3WGWA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.701256312 (26h13m15.484596411s) [noise=true , address=212.102.52.216:4397 , piece_id=5UNX4TPU63Q4AG726PSLVBFWX7ZZJL766ULEHEIIU3S4LG3EUGDA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.735255313 (26h13m15.518595412s) [noise=true , address=212.102.52.213:4331 , piece_id=LTU5YKIGVWFRQXGCECWGW656T6JMUDM7RRK6QD5CRGBB5IKMZXTQ, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.738998335 (26h13m15.522338434s) [noise=true , address=212.102.52.215:4380 , piece_id=6XJ45LCPGAIOCFHI35GODZRUNRS2Z63RF6WIKFUXV5R3GMSH7YJQ, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.739008105 (26h13m15.522348204s) [seg_pos={0 0}, upload=25083]: Satisfied cohort requirements after 16 pieces
May 26 04:17:29.739201553 (26h13m15.522541652s) [noise=true , address=212.102.52.217:4423 , piece_id=2KWQV3P3UYSFMBJ5BJNIHGVNZZXSCD3DEJXUV7SYVDALJSXIJAPA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TgGtX5XRdWH1qn6V5fZpQ8cY5oA2NyZx1AM17XANmr99AG9oh): protocol: expected piece hash; context canceled; EOF
May 26 04:17:29.739208446 (26h13m15.522548545s) [noise=true , address=212.102.52.217:4423 , piece_id=2KWQV3P3UYSFMBJ5BJNIHGVNZZXSCD3DEJXUV7SYVDALJSXIJAPA, seg_pos={0 0}, upload=25083]: Long tail cancellation due to optimal threshold
May 26 04:17:29.739277749 (26h13m15.522617858s) [noise=true , address=212.102.52.213:4343 , piece_id=AUT3LEUM7MPZIL33QGT47V7QOOZTHR3A2W2S6EMKJBY4B3AUA4ZA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129SbAN2aTSXzo8JLELV2ssdkmT5k7kbTcvxSZE4sFsVaKw1Aok): protocol: expected piece hash; context canceled; EOF
May 26 04:17:29.739288459 (26h13m15.522628598s) [noise=true , address=212.102.52.213:4343 , piece_id=AUT3LEUM7MPZIL33QGT47V7QOOZTHR3A2W2S6EMKJBY4B3AUA4ZA, seg_pos={0 0}, upload=25083]: Long tail cancellation due to optimal threshold
May 26 04:17:29.739324538 (26h13m15.522664637s) [noise=true , address=212.102.52.216:4408 , piece_id=23VTNWQ4FKNZ4O552CZQERQ5KMMHSCP4Y46PNSMWU4HGHSDFLIOA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BsdeZy1odVCAgcjxU6KdveXhUUTJKviRrrHLNCUuVergW6zVk): protocol: expected piece hash; context canceled; EOF
May 26 04:17:29.739330078 (26h13m15.522670177s) [noise=true , address=212.102.52.216:4408 , piece_id=23VTNWQ4FKNZ4O552CZQERQ5KMMHSCP4Y46PNSMWU4HGHSDFLIOA, seg_pos={0 0}, upload=25083]: Long tail cancellation due to optimal threshold
May 26 04:17:29.739420791 (26h13m15.52276089s) [noise=true , address=212.102.52.213:4333 , piece_id=O4MDU2BWFAOHKWNQ3KS24KNT6VYDQ4MGLBGCWW4WZZTPMVBT4OCQ, seg_pos={0 0}, upload=25083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Gm3eZxZPxPeHqLQ32kpXMpCgnwRoP8epBYtiFF1EH3U9LEazY): protocol: expected piece hash; context canceled; EOF
May 26 04:17:29.739426021 (26h13m15.52276612s) [noise=true , address=212.102.52.213:4333 , piece_id=O4MDU2BWFAOHKWNQ3KS24KNT6VYDQ4MGLBGCWW4WZZTPMVBT4OCQ, seg_pos={0 0}, upload=25083]: Long tail cancellation due to optimal threshold
May 26 04:17:29.739475536 (26h13m15.522815635s) [seg_pos={0 0}, upload=25083]: Done waiting for segment. successful: 16 optimal: 16 errs: <nil>
May 26 04:17:29.739535340 (26h13m15.522875439s) [upload=25083]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 26 04:17:29.739552654 (26h13m15.522892753s) [upload=25083]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 26 04:17:39.024892360 (26h13m24.808232459s) [upload=25083]: Done uploading segments...
May 26 04:17:39.024959289 (26h13m24.808299388s) [upload=25083]: Upload finished. err: metaclient: context deadline exceeded
Duration: 14.753994409s
core
uplink
uk-select-1
uk1-placement-17
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.X..........................................................] duration:
6.260700091s trace:
https://jaeger.dev.storj.io/trace/2f99fbcd176b1617
FAIL (2026-05-26 04:25:56 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
May 26 03:37:44.554844381 (25h33m30.33818448s) [upload=24452]: Starting upload
May 26 03:37:44.554914775 (25h33m30.338254884s) [upload=24452]: Done starting upload
May 26 03:37:44.555010087 (25h33m30.338350186s) [upload=24452]: Uploading segments...
May 26 03:37:44.555154713 (25h33m30.338494812s) [upload=24452]: Got next segment. Inline: false
May 26 03:37:44.555190892 (25h33m30.338530991s) [upload=24452]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
May 26 03:37:44.555194178 (25h33m30.338534277s) [upload=24452]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
May 26 03:37:44.796393493 (25h33m30.579733593s) [seg_pos={0 0}, upload=24452]: Begin upload segment...
May 26 03:37:44.796434813 (25h33m30.579774902s) [seg_pos={0 0}, upload=24452]: Done begin upload segment.
May 26 03:37:44.796455061 (25h33m30.57979516s) [noise=true , address=145.79.227.26:4657 , piece_id=JCI2K722O5F2HGZR4POW3UMUCGPXQUHMSZFGJL7IL2TSY3T4LKNA, seg_pos={0 0}, upload=24452]: Uploading piece...
May 26 03:37:44.796479167 (25h33m30.579819266s) [noise=true , address=77.90.31.11:4558 , piece_id=SGKGDSAPYDNZ2KDMXJXCQTENMFPAGI6VAXZY3WQUBIZSRFZHPWZQ, seg_pos={0 0}, upload=24452]: Uploading piece...
May 26 03:37:44.796547607 (25h33m30.579887706s) [noise=true , address=145.79.227.15:4873 , piece_id=MCX4ZRVNDVT53WE7W5X7JZ53ZFVOJIUWMZBPHCXAZBUXS4RULCTA, seg_pos={0 0}, upload=24452]: Uploading piece...
May 26 03:37:44.796944846 (25h33m30.580284935s) [noise=true , address=145.79.227.5:4224 , piece_id=QYELKAAUIN72JYBUBRWGKGOF5D4FZ6M4CV2LIRWYMX6Q3FU2TECA, seg_pos={0 0}, upload=24452]: Uploading piece...
May 26 03:37:44.797283402 (25h33m30.580623501s) [noise=true , address=77.90.31.16:4735 , piece_id=EHMPUNQ3X4KQ3OS3Y2ZSAN6QLRGPS2W6DL635MO7IN6FO7QX7LFQ, seg_pos={0 0}, upload=24452]: Uploading piece...
May 26 03:37:44.797625474 (25h33m30.580965573s) [noise=true , address=145.79.227.40:4049 , piece_id=BAAQV3TB3GMDZNLG6UUEMPNJ2R5E5RVSEFBTOPDQHNUXJFB6XOLA, seg_pos={0 0}, upload=24452]: Uploading piece...
May 26 03:37:44.798009868 (25h33m30.581349967s) [noise=true , address=145.79.227.8:4332 , piece_id=LRJG7OY6SCUWAEATGPICV3SQLID25AAPL5DJHMRX4T66MOVT4ARQ, seg_pos={0 0}, upload=24452]: Uploading piece...
May 26 03:37:44.798332895 (25h33m30.581672994s) [noise=true , address=145.79.227.45:4234 , piece_id=2YBD3GONCSIP3A63LSVRI647YC3NTWSXUCBY5ZWEIWMT5B7D7YLQ, seg_pos={0 0}, upload=24452]: Uploading piece...
May 26 03:37:44.798639480 (25h33m30.581979569s) [noise=true , address=145.79.227.10:4401 , piece_id=FDK7SAUVXCQM2XHWJQMLHMLEVYVYH3FWGR4CI5KLFPPFZJRCNJKQ, seg_pos={0 0}, upload=24452]: Uploading piece...
May 26 03:37:44.799006550 (25h33m30.582346649s) [noise=true , address=45.91.116.12:4227 , piece_id=VUZQ6SGI6MGKMHZOT4KZP2Z2CTUGHTQ6373WQRZPTN2KANTEGDTA, seg_pos={0 0}, upload=24452]: Uploading piece...
May 26 03:37:44.799936012 (25h33m30.583276102s) [noise=true , address=45.91.116.4:4947 , piece_id=D2SJUTVUCPKJ2UL2UVZPCGITIXAIMYGDVSSD3Z5XUUN7SB6WAADA, seg_pos={0 0}, upload=24452]: Uploading piece...
May 26 03:37:44.801075256 (25h33m30.584415356s) [noise=true , address=145.79.227.12:4769 , piece_id=4CMAB7IQW3DY72X4R2M4D3GEDZXHQHWYEVBN3VKXMSSYABFOMJWA, seg_pos={0 0}, upload=24452]: Uploading piece...
May 26 03:37:44.802249427 (25h33m30.585589526s) [noise=true , address=145.79.227.21:4473 , piece_id=CV3E5SXKUA3OAW2QRQHHNDSK74MYME6AUNE7YD2YQJMP6MAVSQEQ, seg_pos={0 0}, upload=24452]: Uploading piece...
May 26 03:37:44.803442773 (25h33m30.586782872s) [noise=true , address=145.79.227.49:4340 , piece_id=H4PP2BCFBCRUPINAFKPEIS7IKRCDVILC4EPYEEWNTBSHTNICWGGA, seg_pos={0 0}, upload=24452]: Uploading piece...
May 26 03:37:44.804531642 (25h33m30.587871741s) [noise=true , address=77.90.31.17:4773 , piece_id=45PHIJTVNU5KTCWK4RUCJ3MF2TJSA4B37W4CG6UJ3GFRRPQQ5WZA, seg_pos={0 0}, upload=24452]: Uploading piece...
May 26 03:37:44.805927776 (25h33m30.589267875s) [noise=true , address=77.90.31.12:4589 , piece_id=P6TPHQWR7IOYUKZ4VZKC6CH77YAJJUU2A33WTECMQPUVI2ZCUQYA, seg_pos={0 0}, upload=24452]: Uploading piece...
May 26 03:37:46.520818854 (25h33m32.304158953s) [upload=24452]: Next returned nil segment
May 26 03:37:46.520830196 (25h33m32.304170295s) [upload=24452]: Waiting for error group managing segments...
May 26 03:37:46.802301060 (25h33m32.585641159s) [noise=true , address=145.79.227.8:4332 , piece_id=LRJG7OY6SCUWAEATGPICV3SQLID25AAPL5DJHMRX4T66MOVT4ARQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil>
May 26 03:37:46.805062139 (25h33m32.588402238s) [noise=true , address=145.79.227.12:4769 , piece_id=4CMAB7IQW3DY72X4R2M4D3GEDZXHQHWYEVBN3VKXMSSYABFOMJWA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil>
May 26 03:37:46.818069497 (25h33m32.601409586s) [noise=true , address=145.79.227.49:4340 , piece_id=H4PP2BCFBCRUPINAFKPEIS7IKRCDVILC4EPYEEWNTBSHTNICWGGA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil>
May 26 03:37:46.818763521 (25h33m32.60210362s) [noise=true , address=145.79.227.26:4657 , piece_id=JCI2K722O5F2HGZR4POW3UMUCGPXQUHMSZFGJL7IL2TSY3T4LKNA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil>
May 26 03:37:46.832663342 (25h33m32.616003451s) [noise=true , address=145.79.227.21:4473 , piece_id=CV3E5SXKUA3OAW2QRQHHNDSK74MYME6AUNE7YD2YQJMP6MAVSQEQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil>
May 26 03:37:46.841016390 (25h33m32.624356489s) [noise=true , address=145.79.227.45:4234 , piece_id=2YBD3GONCSIP3A63LSVRI647YC3NTWSXUCBY5ZWEIWMT5B7D7YLQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil>
May 26 03:37:46.842493007 (25h33m32.625833106s) [noise=true , address=145.79.227.40:4049 , piece_id=BAAQV3TB3GMDZNLG6UUEMPNJ2R5E5RVSEFBTOPDQHNUXJFB6XOLA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil>
May 26 03:37:46.855901301 (25h33m32.63924139s) [noise=true , address=145.79.227.5:4224 , piece_id=QYELKAAUIN72JYBUBRWGKGOF5D4FZ6M4CV2LIRWYMX6Q3FU2TECA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil>
May 26 03:37:46.872794464 (25h33m32.656134563s) [noise=true , address=145.79.227.10:4401 , piece_id=FDK7SAUVXCQM2XHWJQMLHMLEVYVYH3FWGR4CI5KLFPPFZJRCNJKQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil>
May 26 03:37:46.904359019 (25h33m32.687699148s) [noise=true , address=145.79.227.15:4873 , piece_id=MCX4ZRVNDVT53WE7W5X7JZ53ZFVOJIUWMZBPHCXAZBUXS4RULCTA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil>
May 26 03:37:47.599588789 (25h33m33.382928888s) [noise=true , address=45.91.116.12:4227 , piece_id=VUZQ6SGI6MGKMHZOT4KZP2Z2CTUGHTQ6373WQRZPTN2KANTEGDTA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil>
May 26 03:37:47.616657837 (25h33m33.399997936s) [noise=true , address=45.91.116.4:4947 , piece_id=D2SJUTVUCPKJ2UL2UVZPCGITIXAIMYGDVSSD3Z5XUUN7SB6WAADA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil>
May 26 03:38:13.890981361 (25h33m59.67432147s) [noise=true , address=77.90.31.12:4589 , piece_id=P6TPHQWR7IOYUKZ4VZKC6CH77YAJJUU2A33WTECMQPUVI2ZCUQYA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: rpc: dial tcp 77.90.31.12:4589: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
May 26 03:38:13.891042047 (25h33m59.674382146s) [noise=true , address=77.90.31.16:4735 , piece_id=EHMPUNQ3X4KQ3OS3Y2ZSAN6QLRGPS2W6DL635MO7IN6FO7QX7LFQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: rpc: dial tcp 77.90.31.16:4735: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
May 26 03:38:13.891153340 (25h33m59.674493439s) [noise=true , address=77.90.31.17:4773 , piece_id=45PHIJTVNU5KTCWK4RUCJ3MF2TJSA4B37W4CG6UJ3GFRRPQQ5WZA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: ecclient: upload failed (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, address:77.90.31.17:4773): protocol: context deadline exceeded; context deadline exceeded
May 26 03:38:13.891203274 (25h33m59.674543393s) [noise=true , address=77.90.31.11:4558 , piece_id=SGKGDSAPYDNZ2KDMXJXCQTENMFPAGI6VAXZY3WQUBIZSRFZHPWZQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: ecclient: upload failed (node:1mGEP1n1icC2gEkyRVy6BeM4vsrgqYWaQeB5rUtxjrWQjyFhTS, address:77.90.31.11:4558): protocol: write tcp 188.166.225.170:40140->77.90.31.11:4558: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
May 26 03:38:13.891231057 (25h33m59.674571156s) [seg_pos={0 0}, upload=24452]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
May 26 03:38:14.130774583 (25h33m59.914114683s) [upload=24452]: Done uploading segments...
May 26 03:38:14.130801014 (25h33m59.914141113s) [upload=24452]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 6.260720119s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [.................X.....................................X....] duration:
3.942395694s trace:
https://jaeger.dev.storj.io/trace/2f0b656030b9f519
FAIL (2026-05-26 04:25:53 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.942413547s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)antiblip:
4 out of 60 is failed (threshold=3) [......X............X..........X................X............] duration:
3.923341084s trace:
https://jaeger.dev.storj.io/trace/20840979a8f7088f
FAIL (2026-05-26 04:25:29 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
25083]: Uploading piece...
May 26 04:17:28.318973560 (26h13m14.10231366s) [noise=true , address=212.102.52.217:4420 , piece_id=PQM3W3VAO5T2BBPQEREGSHGZ3EYBGJ7LFORGCKUY6NTUP2NNE5PA, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.319026442 (26h13m14.102366541s) [noise=true , address=212.102.52.216:4397 , piece_id=5UNX4TPU63Q4AG726PSLVBFWX7ZZJL766ULEHEIIU3S4LG3EUGDA, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.319245079 (26h13m14.102585178s) [noise=true , address=212.102.52.213:4343 , piece_id=AUT3LEUM7MPZIL33QGT47V7QOOZTHR3A2W2S6EMKJBY4B3AUA4ZA, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.319586399 (26h13m14.102926499s) [noise=true , address=212.102.52.216:4402 , piece_id=VNTORDVWCQPSO5C2QUQUI76CBCTFB7TIQSEL7MWCCYRJ22T4WIBA, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.319879910 (26h13m14.103220009s) [noise=true , address=212.102.52.217:4436 , piece_id=SWQCVL47FBVYZFZIQVYNAD4NYDQLBBJL3S2EYVNHGS2Y54RRAN6A, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.320133124 (26h13m14.103473223s) [noise=true , address=212.102.52.215:4379 , piece_id=FBM4P5FPAP6OCVSHW5QYJOJWRXCO63HY2I6K3EYZKL6BRIUMME5Q, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.320357862 (26h13m14.103697961s) [noise=true , address=212.102.52.214:4362 , piece_id=LB7JGSWGNGU7BQX4FZQLVA2IUDQJT7ZP5PFXOQV4NLYXH7JAFRHA, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.320571960 (26h13m14.103912059s) [noise=true , address=212.102.52.216:4408 , piece_id=23VTNWQ4FKNZ4O552CZQERQ5KMMHSCP4Y46PNSMWU4HGHSDFLIOA, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.320588152 (26h13m14.103928251s) [noise=true , address=212.102.52.213:4344 , piece_id=UEGMZ7I63LXBAKRWSFQDHBZBXPCHG2NMZSGW4PFVHFQORZLRMQYA, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.321070803 (26h13m14.104410912s) [noise=true , address=212.102.52.217:4433 , piece_id=FGN6M3NBSA3ZOLHYPT3THZV4THHNFYSI2R6BDB4AWSQZRCZ2M7TQ, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.321436691 (26h13m14.10477679s) [noise=true , address=212.102.52.215:4380 , piece_id=6XJ45LCPGAIOCFHI35GODZRUNRS2Z63RF6WIKFUXV5R3GMSH7YJQ, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.321852404 (26h13m14.105192503s) [noise=true , address=212.102.52.213:4333 , piece_id=O4MDU2BWFAOHKWNQ3KS24KNT6VYDQ4MGLBGCWW4WZZTPMVBT4OCQ, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.322253419 (26h13m14.105593518s) [noise=true , address=212.102.52.214:4347 , piece_id=VPYOK5SVHEPZ36OOB7PYDAAINGIOUACK4FVDZBMJLWKMWC2D4NXQ, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.322652791 (26h13m14.1059929s) [noise=true , address=212.102.52.213:4331 , piece_id=LTU5YKIGVWFRQXGCECWGW656T6JMUDM7RRK6QD5CRGBB5IKMZXTQ, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.324215282 (26h13m14.107555381s) [noise=true , address=212.102.52.214:4349 , piece_id=HH47IVRCTHTDW4RHMJTEXGNB4ZK62NU6LDP5AS4FVXWPFINGHFNQ, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:29.390980339 (26h13m15.174320438s) [upload=25083]: Next returned nil segment
May 26 04:17:29.390993564 (26h13m15.174333663s) [upload=25083]: Waiting for error group managing segments...
May 26 04:17:29.673854379 (26h13m15.457194479s) [noise=true , address=212.102.52.215:4381 , piece_id=5UZ45SYYXIOMUYBHLRRP6NECZCQ5JW2RIK5MQIV2LMYQBVOON4ZA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.674402245 (26h13m15.457742344s) [noise=true , address=212.102.52.217:4433 , piece_id=FGN6M3NBSA3ZOLHYPT3THZV4THHNFYSI2R6BDB4AWSQZRCZ2M7TQ, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.677827470 (26h13m15.461167569s) [noise=true , address=212.102.52.213:4344 , piece_id=UEGMZ7I63LXBAKRWSFQDHBZBXPCHG2NMZSGW4PFVHFQORZLRMQYA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.678368041 (26h13m15.46170814s) [noise=true , address=212.102.52.215:4379 , piece_id=FBM4P5FPAP6OCVSHW5QYJOJWRXCO63HY2I6K3EYZKL6BRIUMME5Q, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.680215035 (26h13m15.463555134s) [noise=true , address=212.102.52.216:4402 , piece_id=VNTORDVWCQPSO5C2QUQUI76CBCTFB7TIQSEL7MWCCYRJ22T4WIBA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.682230351 (26h13m15.46557045s) [noise=true , address=212.102.52.217:4420 , piece_id=PQM3W3VAO5T2BBPQEREGSHGZ3EYBGJ7LFORGCKUY6NTUP2NNE5PA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.685255182 (26h13m15.468595281s) [noise=true , address=212.102.52.216:4405 , piece_id=65IGD4H7EMWEDYW2TFH643JJ5QW2BKHVFZA6D77MCHRYA6T6ESGQ, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.686472445 (26h13m15.469812544s) [noise=true , address=212.102.52.214:4362 , piece_id=LB7JGSWGNGU7BQX4FZQLVA2IUDQJT7ZP5PFXOQV4NLYXH7JAFRHA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.689410080 (26h13m15.472750179s) [noise=true , address=212.102.52.217:4436 , piece_id=SWQCVL47FBVYZFZIQVYNAD4NYDQLBBJL3S2EYVNHGS2Y54RRAN6A, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.689953157 (26h13m15.473293256s) [noise=true , address=212.102.52.214:4355 , piece_id=RKSUNVZS66X4A7ZY7WEUQXSWOTK2H5VAR7TBDICVC7MGVHA7LTGA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.694371046 (26h13m15.477711155s) [noise=true , address=212.102.52.214:4347 , piece_id=VPYOK5SVHEPZ36OOB7PYDAAINGIOUACK4FVDZBMJLWKMWC2D4NXQ, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.698718109 (26h13m15.482058208s) [noise=true , address=212.102.52.214:4349 , piece_id=HH47IVRCTHTDW4RHMJTEXGNB4ZK62NU6LDP5AS4FVXWPFINGHFNQ, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.699290191 (26h13m15.48263029s) [noise=true , address=212.102.52.215:4389 , piece_id=2OTWYJIV6A2646HAW5TWBTSQVT25V2PBKVANLGGK5SWNVHE3WGWA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.701256312 (26h13m15.484596411s) [noise=true , address=212.102.52.216:4397 , piece_id=5UNX4TPU63Q4AG726PSLVBFWX7ZZJL766ULEHEIIU3S4LG3EUGDA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.735255313 (26h13m15.518595412s) [noise=true , address=212.102.52.213:4331 , piece_id=LTU5YKIGVWFRQXGCECWGW656T6JMUDM7RRK6QD5CRGBB5IKMZXTQ, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.738998335 (26h13m15.522338434s) [noise=true , address=212.102.52.215:4380 , piece_id=6XJ45LCPGAIOCFHI35GODZRUNRS2Z63RF6WIKFUXV5R3GMSH7YJQ, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.739008105 (26h13m15.522348204s) [seg_pos={0 0}, upload=25083]: Satisfied cohort requirements after 16 pieces
May 26 04:17:29.739201553 (26h13m15.522541652s) [noise=true , address=212.102.52.217:4423 , piece_id=2KWQV3P3UYSFMBJ5BJNIHGVNZZXSCD3DEJXUV7SYVDALJSXIJAPA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TgGtX5XRdWH1qn6V5fZpQ8cY5oA2NyZx1AM17XANmr99AG9oh): protocol: expected piece hash; context canceled; EOF
May 26 04:17:29.739208446 (26h13m15.522548545s) [noise=true , address=212.102.52.217:4423 , piece_id=2KWQV3P3UYSFMBJ5BJNIHGVNZZXSCD3DEJXUV7SYVDALJSXIJAPA, seg_pos={0 0}, upload=25083]: Long tail cancellation due to optimal threshold
May 26 04:17:29.739277749 (26h13m15.522617858s) [noise=true , address=212.102.52.213:4343 , piece_id=AUT3LEUM7MPZIL33QGT47V7QOOZTHR3A2W2S6EMKJBY4B3AUA4ZA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129SbAN2aTSXzo8JLELV2ssdkmT5k7kbTcvxSZE4sFsVaKw1Aok): protocol: expected piece hash; context canceled; EOF
May 26 04:17:29.739288459 (26h13m15.522628598s) [noise=true , address=212.102.52.213:4343 , piece_id=AUT3LEUM7MPZIL33QGT47V7QOOZTHR3A2W2S6EMKJBY4B3AUA4ZA, seg_pos={0 0}, upload=25083]: Long tail cancellation due to optimal threshold
May 26 04:17:29.739324538 (26h13m15.522664637s) [noise=true , address=212.102.52.216:4408 , piece_id=23VTNWQ4FKNZ4O552CZQERQ5KMMHSCP4Y46PNSMWU4HGHSDFLIOA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BsdeZy1odVCAgcjxU6KdveXhUUTJKviRrrHLNCUuVergW6zVk): protocol: expected piece hash; context canceled; EOF
May 26 04:17:29.739330078 (26h13m15.522670177s) [noise=true , address=212.102.52.216:4408 , piece_id=23VTNWQ4FKNZ4O552CZQERQ5KMMHSCP4Y46PNSMWU4HGHSDFLIOA, seg_pos={0 0}, upload=25083]: Long tail cancellation due to optimal threshold
May 26 04:17:29.739420791 (26h13m15.52276089s) [noise=true , address=212.102.52.213:4333 , piece_id=O4MDU2BWFAOHKWNQ3KS24KNT6VYDQ4MGLBGCWW4WZZTPMVBT4OCQ, seg_pos={0 0}, upload=25083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Gm3eZxZPxPeHqLQ32kpXMpCgnwRoP8epBYtiFF1EH3U9LEazY): protocol: expected piece hash; context canceled; EOF
May 26 04:17:29.739426021 (26h13m15.52276612s) [noise=true , address=212.102.52.213:4333 , piece_id=O4MDU2BWFAOHKWNQ3KS24KNT6VYDQ4MGLBGCWW4WZZTPMVBT4OCQ, seg_pos={0 0}, upload=25083]: Long tail cancellation due to optimal threshold
May 26 04:17:29.739475536 (26h13m15.522815635s) [seg_pos={0 0}, upload=25083]: Done waiting for segment. successful: 16 optimal: 16 errs: <nil>
May 26 04:17:29.739535340 (26h13m15.522875439s) [upload=25083]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 26 04:17:29.739552654 (26h13m15.522892753s) [upload=25083]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 26 04:17:39.024892360 (26h13m24.808232459s) [upload=25083]: Done uploading segments...
May 26 04:17:39.024959289 (26h13m24.808299388s) [upload=25083]: Upload finished. err: metaclient: context deadline exceeded
Duration: 3.923367343s
core
uplink
uk-select-1
uk1-placement-17
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..X.........................................................] duration:
16.020787442s trace:
https://jaeger.dev.storj.io/trace/2f99fbcd176b1617
FAIL (2026-05-26 04:25:05 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
May 26 03:37:44.554844381 (25h33m30.33818448s) [upload=24452]: Starting upload
May 26 03:37:44.554914775 (25h33m30.338254884s) [upload=24452]: Done starting upload
May 26 03:37:44.555010087 (25h33m30.338350186s) [upload=24452]: Uploading segments...
May 26 03:37:44.555154713 (25h33m30.338494812s) [upload=24452]: Got next segment. Inline: false
May 26 03:37:44.555190892 (25h33m30.338530991s) [upload=24452]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
May 26 03:37:44.555194178 (25h33m30.338534277s) [upload=24452]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
May 26 03:37:44.796393493 (25h33m30.579733593s) [seg_pos={0 0}, upload=24452]: Begin upload segment...
May 26 03:37:44.796434813 (25h33m30.579774902s) [seg_pos={0 0}, upload=24452]: Done begin upload segment.
May 26 03:37:44.796455061 (25h33m30.57979516s) [noise=true , address=145.79.227.26:4657 , piece_id=JCI2K722O5F2HGZR4POW3UMUCGPXQUHMSZFGJL7IL2TSY3T4LKNA, seg_pos={0 0}, upload=24452]: Uploading piece...
May 26 03:37:44.796479167 (25h33m30.579819266s) [noise=true , address=77.90.31.11:4558 , piece_id=SGKGDSAPYDNZ2KDMXJXCQTENMFPAGI6VAXZY3WQUBIZSRFZHPWZQ, seg_pos={0 0}, upload=24452]: Uploading piece...
May 26 03:37:44.796547607 (25h33m30.579887706s) [noise=true , address=145.79.227.15:4873 , piece_id=MCX4ZRVNDVT53WE7W5X7JZ53ZFVOJIUWMZBPHCXAZBUXS4RULCTA, seg_pos={0 0}, upload=24452]: Uploading piece...
May 26 03:37:44.796944846 (25h33m30.580284935s) [noise=true , address=145.79.227.5:4224 , piece_id=QYELKAAUIN72JYBUBRWGKGOF5D4FZ6M4CV2LIRWYMX6Q3FU2TECA, seg_pos={0 0}, upload=24452]: Uploading piece...
May 26 03:37:44.797283402 (25h33m30.580623501s) [noise=true , address=77.90.31.16:4735 , piece_id=EHMPUNQ3X4KQ3OS3Y2ZSAN6QLRGPS2W6DL635MO7IN6FO7QX7LFQ, seg_pos={0 0}, upload=24452]: Uploading piece...
May 26 03:37:44.797625474 (25h33m30.580965573s) [noise=true , address=145.79.227.40:4049 , piece_id=BAAQV3TB3GMDZNLG6UUEMPNJ2R5E5RVSEFBTOPDQHNUXJFB6XOLA, seg_pos={0 0}, upload=24452]: Uploading piece...
May 26 03:37:44.798009868 (25h33m30.581349967s) [noise=true , address=145.79.227.8:4332 , piece_id=LRJG7OY6SCUWAEATGPICV3SQLID25AAPL5DJHMRX4T66MOVT4ARQ, seg_pos={0 0}, upload=24452]: Uploading piece...
May 26 03:37:44.798332895 (25h33m30.581672994s) [noise=true , address=145.79.227.45:4234 , piece_id=2YBD3GONCSIP3A63LSVRI647YC3NTWSXUCBY5ZWEIWMT5B7D7YLQ, seg_pos={0 0}, upload=24452]: Uploading piece...
May 26 03:37:44.798639480 (25h33m30.581979569s) [noise=true , address=145.79.227.10:4401 , piece_id=FDK7SAUVXCQM2XHWJQMLHMLEVYVYH3FWGR4CI5KLFPPFZJRCNJKQ, seg_pos={0 0}, upload=24452]: Uploading piece...
May 26 03:37:44.799006550 (25h33m30.582346649s) [noise=true , address=45.91.116.12:4227 , piece_id=VUZQ6SGI6MGKMHZOT4KZP2Z2CTUGHTQ6373WQRZPTN2KANTEGDTA, seg_pos={0 0}, upload=24452]: Uploading piece...
May 26 03:37:44.799936012 (25h33m30.583276102s) [noise=true , address=45.91.116.4:4947 , piece_id=D2SJUTVUCPKJ2UL2UVZPCGITIXAIMYGDVSSD3Z5XUUN7SB6WAADA, seg_pos={0 0}, upload=24452]: Uploading piece...
May 26 03:37:44.801075256 (25h33m30.584415356s) [noise=true , address=145.79.227.12:4769 , piece_id=4CMAB7IQW3DY72X4R2M4D3GEDZXHQHWYEVBN3VKXMSSYABFOMJWA, seg_pos={0 0}, upload=24452]: Uploading piece...
May 26 03:37:44.802249427 (25h33m30.585589526s) [noise=true , address=145.79.227.21:4473 , piece_id=CV3E5SXKUA3OAW2QRQHHNDSK74MYME6AUNE7YD2YQJMP6MAVSQEQ, seg_pos={0 0}, upload=24452]: Uploading piece...
May 26 03:37:44.803442773 (25h33m30.586782872s) [noise=true , address=145.79.227.49:4340 , piece_id=H4PP2BCFBCRUPINAFKPEIS7IKRCDVILC4EPYEEWNTBSHTNICWGGA, seg_pos={0 0}, upload=24452]: Uploading piece...
May 26 03:37:44.804531642 (25h33m30.587871741s) [noise=true , address=77.90.31.17:4773 , piece_id=45PHIJTVNU5KTCWK4RUCJ3MF2TJSA4B37W4CG6UJ3GFRRPQQ5WZA, seg_pos={0 0}, upload=24452]: Uploading piece...
May 26 03:37:44.805927776 (25h33m30.589267875s) [noise=true , address=77.90.31.12:4589 , piece_id=P6TPHQWR7IOYUKZ4VZKC6CH77YAJJUU2A33WTECMQPUVI2ZCUQYA, seg_pos={0 0}, upload=24452]: Uploading piece...
May 26 03:37:46.520818854 (25h33m32.304158953s) [upload=24452]: Next returned nil segment
May 26 03:37:46.520830196 (25h33m32.304170295s) [upload=24452]: Waiting for error group managing segments...
May 26 03:37:46.802301060 (25h33m32.585641159s) [noise=true , address=145.79.227.8:4332 , piece_id=LRJG7OY6SCUWAEATGPICV3SQLID25AAPL5DJHMRX4T66MOVT4ARQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil>
May 26 03:37:46.805062139 (25h33m32.588402238s) [noise=true , address=145.79.227.12:4769 , piece_id=4CMAB7IQW3DY72X4R2M4D3GEDZXHQHWYEVBN3VKXMSSYABFOMJWA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil>
May 26 03:37:46.818069497 (25h33m32.601409586s) [noise=true , address=145.79.227.49:4340 , piece_id=H4PP2BCFBCRUPINAFKPEIS7IKRCDVILC4EPYEEWNTBSHTNICWGGA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil>
May 26 03:37:46.818763521 (25h33m32.60210362s) [noise=true , address=145.79.227.26:4657 , piece_id=JCI2K722O5F2HGZR4POW3UMUCGPXQUHMSZFGJL7IL2TSY3T4LKNA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil>
May 26 03:37:46.832663342 (25h33m32.616003451s) [noise=true , address=145.79.227.21:4473 , piece_id=CV3E5SXKUA3OAW2QRQHHNDSK74MYME6AUNE7YD2YQJMP6MAVSQEQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil>
May 26 03:37:46.841016390 (25h33m32.624356489s) [noise=true , address=145.79.227.45:4234 , piece_id=2YBD3GONCSIP3A63LSVRI647YC3NTWSXUCBY5ZWEIWMT5B7D7YLQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil>
May 26 03:37:46.842493007 (25h33m32.625833106s) [noise=true , address=145.79.227.40:4049 , piece_id=BAAQV3TB3GMDZNLG6UUEMPNJ2R5E5RVSEFBTOPDQHNUXJFB6XOLA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil>
May 26 03:37:46.855901301 (25h33m32.63924139s) [noise=true , address=145.79.227.5:4224 , piece_id=QYELKAAUIN72JYBUBRWGKGOF5D4FZ6M4CV2LIRWYMX6Q3FU2TECA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil>
May 26 03:37:46.872794464 (25h33m32.656134563s) [noise=true , address=145.79.227.10:4401 , piece_id=FDK7SAUVXCQM2XHWJQMLHMLEVYVYH3FWGR4CI5KLFPPFZJRCNJKQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil>
May 26 03:37:46.904359019 (25h33m32.687699148s) [noise=true , address=145.79.227.15:4873 , piece_id=MCX4ZRVNDVT53WE7W5X7JZ53ZFVOJIUWMZBPHCXAZBUXS4RULCTA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil>
May 26 03:37:47.599588789 (25h33m33.382928888s) [noise=true , address=45.91.116.12:4227 , piece_id=VUZQ6SGI6MGKMHZOT4KZP2Z2CTUGHTQ6373WQRZPTN2KANTEGDTA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil>
May 26 03:37:47.616657837 (25h33m33.399997936s) [noise=true , address=45.91.116.4:4947 , piece_id=D2SJUTVUCPKJ2UL2UVZPCGITIXAIMYGDVSSD3Z5XUUN7SB6WAADA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil>
May 26 03:38:13.890981361 (25h33m59.67432147s) [noise=true , address=77.90.31.12:4589 , piece_id=P6TPHQWR7IOYUKZ4VZKC6CH77YAJJUU2A33WTECMQPUVI2ZCUQYA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: rpc: dial tcp 77.90.31.12:4589: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
May 26 03:38:13.891042047 (25h33m59.674382146s) [noise=true , address=77.90.31.16:4735 , piece_id=EHMPUNQ3X4KQ3OS3Y2ZSAN6QLRGPS2W6DL635MO7IN6FO7QX7LFQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: rpc: dial tcp 77.90.31.16:4735: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
May 26 03:38:13.891153340 (25h33m59.674493439s) [noise=true , address=77.90.31.17:4773 , piece_id=45PHIJTVNU5KTCWK4RUCJ3MF2TJSA4B37W4CG6UJ3GFRRPQQ5WZA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: ecclient: upload failed (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, address:77.90.31.17:4773): protocol: context deadline exceeded; context deadline exceeded
May 26 03:38:13.891203274 (25h33m59.674543393s) [noise=true , address=77.90.31.11:4558 , piece_id=SGKGDSAPYDNZ2KDMXJXCQTENMFPAGI6VAXZY3WQUBIZSRFZHPWZQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: ecclient: upload failed (node:1mGEP1n1icC2gEkyRVy6BeM4vsrgqYWaQeB5rUtxjrWQjyFhTS, address:77.90.31.11:4558): protocol: write tcp 188.166.225.170:40140->77.90.31.11:4558: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
May 26 03:38:13.891231057 (25h33m59.674571156s) [seg_pos={0 0}, upload=24452]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
May 26 03:38:14.130774583 (25h33m59.914114683s) [upload=24452]: Done uploading segments...
May 26 03:38:14.130801014 (25h33m59.914141113s) [upload=24452]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 16.02080726s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)antiblip:
4 out of 60 is failed (threshold=3) [.......X............X..........X................X...........] duration:
4.034165623s trace:
https://jaeger.dev.storj.io/trace/20840979a8f7088f
FAIL (2026-05-26 04:24:59 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
25083]: Uploading piece...
May 26 04:17:28.318973560 (26h13m14.10231366s) [noise=true , address=212.102.52.217:4420 , piece_id=PQM3W3VAO5T2BBPQEREGSHGZ3EYBGJ7LFORGCKUY6NTUP2NNE5PA, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.319026442 (26h13m14.102366541s) [noise=true , address=212.102.52.216:4397 , piece_id=5UNX4TPU63Q4AG726PSLVBFWX7ZZJL766ULEHEIIU3S4LG3EUGDA, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.319245079 (26h13m14.102585178s) [noise=true , address=212.102.52.213:4343 , piece_id=AUT3LEUM7MPZIL33QGT47V7QOOZTHR3A2W2S6EMKJBY4B3AUA4ZA, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.319586399 (26h13m14.102926499s) [noise=true , address=212.102.52.216:4402 , piece_id=VNTORDVWCQPSO5C2QUQUI76CBCTFB7TIQSEL7MWCCYRJ22T4WIBA, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.319879910 (26h13m14.103220009s) [noise=true , address=212.102.52.217:4436 , piece_id=SWQCVL47FBVYZFZIQVYNAD4NYDQLBBJL3S2EYVNHGS2Y54RRAN6A, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.320133124 (26h13m14.103473223s) [noise=true , address=212.102.52.215:4379 , piece_id=FBM4P5FPAP6OCVSHW5QYJOJWRXCO63HY2I6K3EYZKL6BRIUMME5Q, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.320357862 (26h13m14.103697961s) [noise=true , address=212.102.52.214:4362 , piece_id=LB7JGSWGNGU7BQX4FZQLVA2IUDQJT7ZP5PFXOQV4NLYXH7JAFRHA, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.320571960 (26h13m14.103912059s) [noise=true , address=212.102.52.216:4408 , piece_id=23VTNWQ4FKNZ4O552CZQERQ5KMMHSCP4Y46PNSMWU4HGHSDFLIOA, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.320588152 (26h13m14.103928251s) [noise=true , address=212.102.52.213:4344 , piece_id=UEGMZ7I63LXBAKRWSFQDHBZBXPCHG2NMZSGW4PFVHFQORZLRMQYA, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.321070803 (26h13m14.104410912s) [noise=true , address=212.102.52.217:4433 , piece_id=FGN6M3NBSA3ZOLHYPT3THZV4THHNFYSI2R6BDB4AWSQZRCZ2M7TQ, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.321436691 (26h13m14.10477679s) [noise=true , address=212.102.52.215:4380 , piece_id=6XJ45LCPGAIOCFHI35GODZRUNRS2Z63RF6WIKFUXV5R3GMSH7YJQ, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.321852404 (26h13m14.105192503s) [noise=true , address=212.102.52.213:4333 , piece_id=O4MDU2BWFAOHKWNQ3KS24KNT6VYDQ4MGLBGCWW4WZZTPMVBT4OCQ, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.322253419 (26h13m14.105593518s) [noise=true , address=212.102.52.214:4347 , piece_id=VPYOK5SVHEPZ36OOB7PYDAAINGIOUACK4FVDZBMJLWKMWC2D4NXQ, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.322652791 (26h13m14.1059929s) [noise=true , address=212.102.52.213:4331 , piece_id=LTU5YKIGVWFRQXGCECWGW656T6JMUDM7RRK6QD5CRGBB5IKMZXTQ, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.324215282 (26h13m14.107555381s) [noise=true , address=212.102.52.214:4349 , piece_id=HH47IVRCTHTDW4RHMJTEXGNB4ZK62NU6LDP5AS4FVXWPFINGHFNQ, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:29.390980339 (26h13m15.174320438s) [upload=25083]: Next returned nil segment
May 26 04:17:29.390993564 (26h13m15.174333663s) [upload=25083]: Waiting for error group managing segments...
May 26 04:17:29.673854379 (26h13m15.457194479s) [noise=true , address=212.102.52.215:4381 , piece_id=5UZ45SYYXIOMUYBHLRRP6NECZCQ5JW2RIK5MQIV2LMYQBVOON4ZA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.674402245 (26h13m15.457742344s) [noise=true , address=212.102.52.217:4433 , piece_id=FGN6M3NBSA3ZOLHYPT3THZV4THHNFYSI2R6BDB4AWSQZRCZ2M7TQ, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.677827470 (26h13m15.461167569s) [noise=true , address=212.102.52.213:4344 , piece_id=UEGMZ7I63LXBAKRWSFQDHBZBXPCHG2NMZSGW4PFVHFQORZLRMQYA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.678368041 (26h13m15.46170814s) [noise=true , address=212.102.52.215:4379 , piece_id=FBM4P5FPAP6OCVSHW5QYJOJWRXCO63HY2I6K3EYZKL6BRIUMME5Q, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.680215035 (26h13m15.463555134s) [noise=true , address=212.102.52.216:4402 , piece_id=VNTORDVWCQPSO5C2QUQUI76CBCTFB7TIQSEL7MWCCYRJ22T4WIBA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.682230351 (26h13m15.46557045s) [noise=true , address=212.102.52.217:4420 , piece_id=PQM3W3VAO5T2BBPQEREGSHGZ3EYBGJ7LFORGCKUY6NTUP2NNE5PA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.685255182 (26h13m15.468595281s) [noise=true , address=212.102.52.216:4405 , piece_id=65IGD4H7EMWEDYW2TFH643JJ5QW2BKHVFZA6D77MCHRYA6T6ESGQ, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.686472445 (26h13m15.469812544s) [noise=true , address=212.102.52.214:4362 , piece_id=LB7JGSWGNGU7BQX4FZQLVA2IUDQJT7ZP5PFXOQV4NLYXH7JAFRHA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.689410080 (26h13m15.472750179s) [noise=true , address=212.102.52.217:4436 , piece_id=SWQCVL47FBVYZFZIQVYNAD4NYDQLBBJL3S2EYVNHGS2Y54RRAN6A, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.689953157 (26h13m15.473293256s) [noise=true , address=212.102.52.214:4355 , piece_id=RKSUNVZS66X4A7ZY7WEUQXSWOTK2H5VAR7TBDICVC7MGVHA7LTGA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.694371046 (26h13m15.477711155s) [noise=true , address=212.102.52.214:4347 , piece_id=VPYOK5SVHEPZ36OOB7PYDAAINGIOUACK4FVDZBMJLWKMWC2D4NXQ, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.698718109 (26h13m15.482058208s) [noise=true , address=212.102.52.214:4349 , piece_id=HH47IVRCTHTDW4RHMJTEXGNB4ZK62NU6LDP5AS4FVXWPFINGHFNQ, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.699290191 (26h13m15.48263029s) [noise=true , address=212.102.52.215:4389 , piece_id=2OTWYJIV6A2646HAW5TWBTSQVT25V2PBKVANLGGK5SWNVHE3WGWA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.701256312 (26h13m15.484596411s) [noise=true , address=212.102.52.216:4397 , piece_id=5UNX4TPU63Q4AG726PSLVBFWX7ZZJL766ULEHEIIU3S4LG3EUGDA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.735255313 (26h13m15.518595412s) [noise=true , address=212.102.52.213:4331 , piece_id=LTU5YKIGVWFRQXGCECWGW656T6JMUDM7RRK6QD5CRGBB5IKMZXTQ, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.738998335 (26h13m15.522338434s) [noise=true , address=212.102.52.215:4380 , piece_id=6XJ45LCPGAIOCFHI35GODZRUNRS2Z63RF6WIKFUXV5R3GMSH7YJQ, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.739008105 (26h13m15.522348204s) [seg_pos={0 0}, upload=25083]: Satisfied cohort requirements after 16 pieces
May 26 04:17:29.739201553 (26h13m15.522541652s) [noise=true , address=212.102.52.217:4423 , piece_id=2KWQV3P3UYSFMBJ5BJNIHGVNZZXSCD3DEJXUV7SYVDALJSXIJAPA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TgGtX5XRdWH1qn6V5fZpQ8cY5oA2NyZx1AM17XANmr99AG9oh): protocol: expected piece hash; context canceled; EOF
May 26 04:17:29.739208446 (26h13m15.522548545s) [noise=true , address=212.102.52.217:4423 , piece_id=2KWQV3P3UYSFMBJ5BJNIHGVNZZXSCD3DEJXUV7SYVDALJSXIJAPA, seg_pos={0 0}, upload=25083]: Long tail cancellation due to optimal threshold
May 26 04:17:29.739277749 (26h13m15.522617858s) [noise=true , address=212.102.52.213:4343 , piece_id=AUT3LEUM7MPZIL33QGT47V7QOOZTHR3A2W2S6EMKJBY4B3AUA4ZA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129SbAN2aTSXzo8JLELV2ssdkmT5k7kbTcvxSZE4sFsVaKw1Aok): protocol: expected piece hash; context canceled; EOF
May 26 04:17:29.739288459 (26h13m15.522628598s) [noise=true , address=212.102.52.213:4343 , piece_id=AUT3LEUM7MPZIL33QGT47V7QOOZTHR3A2W2S6EMKJBY4B3AUA4ZA, seg_pos={0 0}, upload=25083]: Long tail cancellation due to optimal threshold
May 26 04:17:29.739324538 (26h13m15.522664637s) [noise=true , address=212.102.52.216:4408 , piece_id=23VTNWQ4FKNZ4O552CZQERQ5KMMHSCP4Y46PNSMWU4HGHSDFLIOA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BsdeZy1odVCAgcjxU6KdveXhUUTJKviRrrHLNCUuVergW6zVk): protocol: expected piece hash; context canceled; EOF
May 26 04:17:29.739330078 (26h13m15.522670177s) [noise=true , address=212.102.52.216:4408 , piece_id=23VTNWQ4FKNZ4O552CZQERQ5KMMHSCP4Y46PNSMWU4HGHSDFLIOA, seg_pos={0 0}, upload=25083]: Long tail cancellation due to optimal threshold
May 26 04:17:29.739420791 (26h13m15.52276089s) [noise=true , address=212.102.52.213:4333 , piece_id=O4MDU2BWFAOHKWNQ3KS24KNT6VYDQ4MGLBGCWW4WZZTPMVBT4OCQ, seg_pos={0 0}, upload=25083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Gm3eZxZPxPeHqLQ32kpXMpCgnwRoP8epBYtiFF1EH3U9LEazY): protocol: expected piece hash; context canceled; EOF
May 26 04:17:29.739426021 (26h13m15.52276612s) [noise=true , address=212.102.52.213:4333 , piece_id=O4MDU2BWFAOHKWNQ3KS24KNT6VYDQ4MGLBGCWW4WZZTPMVBT4OCQ, seg_pos={0 0}, upload=25083]: Long tail cancellation due to optimal threshold
May 26 04:17:29.739475536 (26h13m15.522815635s) [seg_pos={0 0}, upload=25083]: Done waiting for segment. successful: 16 optimal: 16 errs: <nil>
May 26 04:17:29.739535340 (26h13m15.522875439s) [upload=25083]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 26 04:17:29.739552654 (26h13m15.522892753s) [upload=25083]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 26 04:17:39.024892360 (26h13m24.808232459s) [upload=25083]: Done uploading segments...
May 26 04:17:39.024959289 (26h13m24.808299388s) [upload=25083]: Upload finished. err: metaclient: context deadline exceeded
Duration: 4.034186602s
core
uplink
uk-select-1
uk1-placement-17
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [..................X.....................................X...] duration:
3.758909364s trace:
https://jaeger.dev.storj.io/trace/2f0b656030b9f519
FAIL (2026-05-26 04:24:48 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.758929543s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)antiblip:
4 out of 60 is failed (threshold=3) [........X............X..........X................X..........] duration:
6.150042607s trace:
https://jaeger.dev.storj.io/trace/20840979a8f7088f
FAIL (2026-05-26 04:23:57 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
25083]: Uploading piece...
May 26 04:17:28.318973560 (26h13m14.10231366s) [noise=true , address=212.102.52.217:4420 , piece_id=PQM3W3VAO5T2BBPQEREGSHGZ3EYBGJ7LFORGCKUY6NTUP2NNE5PA, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.319026442 (26h13m14.102366541s) [noise=true , address=212.102.52.216:4397 , piece_id=5UNX4TPU63Q4AG726PSLVBFWX7ZZJL766ULEHEIIU3S4LG3EUGDA, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.319245079 (26h13m14.102585178s) [noise=true , address=212.102.52.213:4343 , piece_id=AUT3LEUM7MPZIL33QGT47V7QOOZTHR3A2W2S6EMKJBY4B3AUA4ZA, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.319586399 (26h13m14.102926499s) [noise=true , address=212.102.52.216:4402 , piece_id=VNTORDVWCQPSO5C2QUQUI76CBCTFB7TIQSEL7MWCCYRJ22T4WIBA, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.319879910 (26h13m14.103220009s) [noise=true , address=212.102.52.217:4436 , piece_id=SWQCVL47FBVYZFZIQVYNAD4NYDQLBBJL3S2EYVNHGS2Y54RRAN6A, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.320133124 (26h13m14.103473223s) [noise=true , address=212.102.52.215:4379 , piece_id=FBM4P5FPAP6OCVSHW5QYJOJWRXCO63HY2I6K3EYZKL6BRIUMME5Q, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.320357862 (26h13m14.103697961s) [noise=true , address=212.102.52.214:4362 , piece_id=LB7JGSWGNGU7BQX4FZQLVA2IUDQJT7ZP5PFXOQV4NLYXH7JAFRHA, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.320571960 (26h13m14.103912059s) [noise=true , address=212.102.52.216:4408 , piece_id=23VTNWQ4FKNZ4O552CZQERQ5KMMHSCP4Y46PNSMWU4HGHSDFLIOA, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.320588152 (26h13m14.103928251s) [noise=true , address=212.102.52.213:4344 , piece_id=UEGMZ7I63LXBAKRWSFQDHBZBXPCHG2NMZSGW4PFVHFQORZLRMQYA, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.321070803 (26h13m14.104410912s) [noise=true , address=212.102.52.217:4433 , piece_id=FGN6M3NBSA3ZOLHYPT3THZV4THHNFYSI2R6BDB4AWSQZRCZ2M7TQ, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.321436691 (26h13m14.10477679s) [noise=true , address=212.102.52.215:4380 , piece_id=6XJ45LCPGAIOCFHI35GODZRUNRS2Z63RF6WIKFUXV5R3GMSH7YJQ, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.321852404 (26h13m14.105192503s) [noise=true , address=212.102.52.213:4333 , piece_id=O4MDU2BWFAOHKWNQ3KS24KNT6VYDQ4MGLBGCWW4WZZTPMVBT4OCQ, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.322253419 (26h13m14.105593518s) [noise=true , address=212.102.52.214:4347 , piece_id=VPYOK5SVHEPZ36OOB7PYDAAINGIOUACK4FVDZBMJLWKMWC2D4NXQ, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.322652791 (26h13m14.1059929s) [noise=true , address=212.102.52.213:4331 , piece_id=LTU5YKIGVWFRQXGCECWGW656T6JMUDM7RRK6QD5CRGBB5IKMZXTQ, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.324215282 (26h13m14.107555381s) [noise=true , address=212.102.52.214:4349 , piece_id=HH47IVRCTHTDW4RHMJTEXGNB4ZK62NU6LDP5AS4FVXWPFINGHFNQ, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:29.390980339 (26h13m15.174320438s) [upload=25083]: Next returned nil segment
May 26 04:17:29.390993564 (26h13m15.174333663s) [upload=25083]: Waiting for error group managing segments...
May 26 04:17:29.673854379 (26h13m15.457194479s) [noise=true , address=212.102.52.215:4381 , piece_id=5UZ45SYYXIOMUYBHLRRP6NECZCQ5JW2RIK5MQIV2LMYQBVOON4ZA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.674402245 (26h13m15.457742344s) [noise=true , address=212.102.52.217:4433 , piece_id=FGN6M3NBSA3ZOLHYPT3THZV4THHNFYSI2R6BDB4AWSQZRCZ2M7TQ, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.677827470 (26h13m15.461167569s) [noise=true , address=212.102.52.213:4344 , piece_id=UEGMZ7I63LXBAKRWSFQDHBZBXPCHG2NMZSGW4PFVHFQORZLRMQYA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.678368041 (26h13m15.46170814s) [noise=true , address=212.102.52.215:4379 , piece_id=FBM4P5FPAP6OCVSHW5QYJOJWRXCO63HY2I6K3EYZKL6BRIUMME5Q, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.680215035 (26h13m15.463555134s) [noise=true , address=212.102.52.216:4402 , piece_id=VNTORDVWCQPSO5C2QUQUI76CBCTFB7TIQSEL7MWCCYRJ22T4WIBA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.682230351 (26h13m15.46557045s) [noise=true , address=212.102.52.217:4420 , piece_id=PQM3W3VAO5T2BBPQEREGSHGZ3EYBGJ7LFORGCKUY6NTUP2NNE5PA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.685255182 (26h13m15.468595281s) [noise=true , address=212.102.52.216:4405 , piece_id=65IGD4H7EMWEDYW2TFH643JJ5QW2BKHVFZA6D77MCHRYA6T6ESGQ, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.686472445 (26h13m15.469812544s) [noise=true , address=212.102.52.214:4362 , piece_id=LB7JGSWGNGU7BQX4FZQLVA2IUDQJT7ZP5PFXOQV4NLYXH7JAFRHA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.689410080 (26h13m15.472750179s) [noise=true , address=212.102.52.217:4436 , piece_id=SWQCVL47FBVYZFZIQVYNAD4NYDQLBBJL3S2EYVNHGS2Y54RRAN6A, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.689953157 (26h13m15.473293256s) [noise=true , address=212.102.52.214:4355 , piece_id=RKSUNVZS66X4A7ZY7WEUQXSWOTK2H5VAR7TBDICVC7MGVHA7LTGA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.694371046 (26h13m15.477711155s) [noise=true , address=212.102.52.214:4347 , piece_id=VPYOK5SVHEPZ36OOB7PYDAAINGIOUACK4FVDZBMJLWKMWC2D4NXQ, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.698718109 (26h13m15.482058208s) [noise=true , address=212.102.52.214:4349 , piece_id=HH47IVRCTHTDW4RHMJTEXGNB4ZK62NU6LDP5AS4FVXWPFINGHFNQ, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.699290191 (26h13m15.48263029s) [noise=true , address=212.102.52.215:4389 , piece_id=2OTWYJIV6A2646HAW5TWBTSQVT25V2PBKVANLGGK5SWNVHE3WGWA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.701256312 (26h13m15.484596411s) [noise=true , address=212.102.52.216:4397 , piece_id=5UNX4TPU63Q4AG726PSLVBFWX7ZZJL766ULEHEIIU3S4LG3EUGDA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.735255313 (26h13m15.518595412s) [noise=true , address=212.102.52.213:4331 , piece_id=LTU5YKIGVWFRQXGCECWGW656T6JMUDM7RRK6QD5CRGBB5IKMZXTQ, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.738998335 (26h13m15.522338434s) [noise=true , address=212.102.52.215:4380 , piece_id=6XJ45LCPGAIOCFHI35GODZRUNRS2Z63RF6WIKFUXV5R3GMSH7YJQ, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.739008105 (26h13m15.522348204s) [seg_pos={0 0}, upload=25083]: Satisfied cohort requirements after 16 pieces
May 26 04:17:29.739201553 (26h13m15.522541652s) [noise=true , address=212.102.52.217:4423 , piece_id=2KWQV3P3UYSFMBJ5BJNIHGVNZZXSCD3DEJXUV7SYVDALJSXIJAPA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TgGtX5XRdWH1qn6V5fZpQ8cY5oA2NyZx1AM17XANmr99AG9oh): protocol: expected piece hash; context canceled; EOF
May 26 04:17:29.739208446 (26h13m15.522548545s) [noise=true , address=212.102.52.217:4423 , piece_id=2KWQV3P3UYSFMBJ5BJNIHGVNZZXSCD3DEJXUV7SYVDALJSXIJAPA, seg_pos={0 0}, upload=25083]: Long tail cancellation due to optimal threshold
May 26 04:17:29.739277749 (26h13m15.522617858s) [noise=true , address=212.102.52.213:4343 , piece_id=AUT3LEUM7MPZIL33QGT47V7QOOZTHR3A2W2S6EMKJBY4B3AUA4ZA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129SbAN2aTSXzo8JLELV2ssdkmT5k7kbTcvxSZE4sFsVaKw1Aok): protocol: expected piece hash; context canceled; EOF
May 26 04:17:29.739288459 (26h13m15.522628598s) [noise=true , address=212.102.52.213:4343 , piece_id=AUT3LEUM7MPZIL33QGT47V7QOOZTHR3A2W2S6EMKJBY4B3AUA4ZA, seg_pos={0 0}, upload=25083]: Long tail cancellation due to optimal threshold
May 26 04:17:29.739324538 (26h13m15.522664637s) [noise=true , address=212.102.52.216:4408 , piece_id=23VTNWQ4FKNZ4O552CZQERQ5KMMHSCP4Y46PNSMWU4HGHSDFLIOA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BsdeZy1odVCAgcjxU6KdveXhUUTJKviRrrHLNCUuVergW6zVk): protocol: expected piece hash; context canceled; EOF
May 26 04:17:29.739330078 (26h13m15.522670177s) [noise=true , address=212.102.52.216:4408 , piece_id=23VTNWQ4FKNZ4O552CZQERQ5KMMHSCP4Y46PNSMWU4HGHSDFLIOA, seg_pos={0 0}, upload=25083]: Long tail cancellation due to optimal threshold
May 26 04:17:29.739420791 (26h13m15.52276089s) [noise=true , address=212.102.52.213:4333 , piece_id=O4MDU2BWFAOHKWNQ3KS24KNT6VYDQ4MGLBGCWW4WZZTPMVBT4OCQ, seg_pos={0 0}, upload=25083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Gm3eZxZPxPeHqLQ32kpXMpCgnwRoP8epBYtiFF1EH3U9LEazY): protocol: expected piece hash; context canceled; EOF
May 26 04:17:29.739426021 (26h13m15.52276612s) [noise=true , address=212.102.52.213:4333 , piece_id=O4MDU2BWFAOHKWNQ3KS24KNT6VYDQ4MGLBGCWW4WZZTPMVBT4OCQ, seg_pos={0 0}, upload=25083]: Long tail cancellation due to optimal threshold
May 26 04:17:29.739475536 (26h13m15.522815635s) [seg_pos={0 0}, upload=25083]: Done waiting for segment. successful: 16 optimal: 16 errs: <nil>
May 26 04:17:29.739535340 (26h13m15.522875439s) [upload=25083]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 26 04:17:29.739552654 (26h13m15.522892753s) [upload=25083]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 26 04:17:39.024892360 (26h13m24.808232459s) [upload=25083]: Done uploading segments...
May 26 04:17:39.024959289 (26h13m24.808299388s) [upload=25083]: Upload finished. err: metaclient: context deadline exceeded
Duration: 6.15006429s
core
uplink
uk-select-1
uk1-placement-17
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...X........................................................] duration:
5.736061416s trace:
https://jaeger.dev.storj.io/trace/2f99fbcd176b1617
FAIL (2026-05-26 04:23:44 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
May 26 03:37:44.554844381 (25h33m30.33818448s) [upload=24452]: Starting upload
May 26 03:37:44.554914775 (25h33m30.338254884s) [upload=24452]: Done starting upload
May 26 03:37:44.555010087 (25h33m30.338350186s) [upload=24452]: Uploading segments...
May 26 03:37:44.555154713 (25h33m30.338494812s) [upload=24452]: Got next segment. Inline: false
May 26 03:37:44.555190892 (25h33m30.338530991s) [upload=24452]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
May 26 03:37:44.555194178 (25h33m30.338534277s) [upload=24452]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
May 26 03:37:44.796393493 (25h33m30.579733593s) [seg_pos={0 0}, upload=24452]: Begin upload segment...
May 26 03:37:44.796434813 (25h33m30.579774902s) [seg_pos={0 0}, upload=24452]: Done begin upload segment.
May 26 03:37:44.796455061 (25h33m30.57979516s) [noise=true , address=145.79.227.26:4657 , piece_id=JCI2K722O5F2HGZR4POW3UMUCGPXQUHMSZFGJL7IL2TSY3T4LKNA, seg_pos={0 0}, upload=24452]: Uploading piece...
May 26 03:37:44.796479167 (25h33m30.579819266s) [noise=true , address=77.90.31.11:4558 , piece_id=SGKGDSAPYDNZ2KDMXJXCQTENMFPAGI6VAXZY3WQUBIZSRFZHPWZQ, seg_pos={0 0}, upload=24452]: Uploading piece...
May 26 03:37:44.796547607 (25h33m30.579887706s) [noise=true , address=145.79.227.15:4873 , piece_id=MCX4ZRVNDVT53WE7W5X7JZ53ZFVOJIUWMZBPHCXAZBUXS4RULCTA, seg_pos={0 0}, upload=24452]: Uploading piece...
May 26 03:37:44.796944846 (25h33m30.580284935s) [noise=true , address=145.79.227.5:4224 , piece_id=QYELKAAUIN72JYBUBRWGKGOF5D4FZ6M4CV2LIRWYMX6Q3FU2TECA, seg_pos={0 0}, upload=24452]: Uploading piece...
May 26 03:37:44.797283402 (25h33m30.580623501s) [noise=true , address=77.90.31.16:4735 , piece_id=EHMPUNQ3X4KQ3OS3Y2ZSAN6QLRGPS2W6DL635MO7IN6FO7QX7LFQ, seg_pos={0 0}, upload=24452]: Uploading piece...
May 26 03:37:44.797625474 (25h33m30.580965573s) [noise=true , address=145.79.227.40:4049 , piece_id=BAAQV3TB3GMDZNLG6UUEMPNJ2R5E5RVSEFBTOPDQHNUXJFB6XOLA, seg_pos={0 0}, upload=24452]: Uploading piece...
May 26 03:37:44.798009868 (25h33m30.581349967s) [noise=true , address=145.79.227.8:4332 , piece_id=LRJG7OY6SCUWAEATGPICV3SQLID25AAPL5DJHMRX4T66MOVT4ARQ, seg_pos={0 0}, upload=24452]: Uploading piece...
May 26 03:37:44.798332895 (25h33m30.581672994s) [noise=true , address=145.79.227.45:4234 , piece_id=2YBD3GONCSIP3A63LSVRI647YC3NTWSXUCBY5ZWEIWMT5B7D7YLQ, seg_pos={0 0}, upload=24452]: Uploading piece...
May 26 03:37:44.798639480 (25h33m30.581979569s) [noise=true , address=145.79.227.10:4401 , piece_id=FDK7SAUVXCQM2XHWJQMLHMLEVYVYH3FWGR4CI5KLFPPFZJRCNJKQ, seg_pos={0 0}, upload=24452]: Uploading piece...
May 26 03:37:44.799006550 (25h33m30.582346649s) [noise=true , address=45.91.116.12:4227 , piece_id=VUZQ6SGI6MGKMHZOT4KZP2Z2CTUGHTQ6373WQRZPTN2KANTEGDTA, seg_pos={0 0}, upload=24452]: Uploading piece...
May 26 03:37:44.799936012 (25h33m30.583276102s) [noise=true , address=45.91.116.4:4947 , piece_id=D2SJUTVUCPKJ2UL2UVZPCGITIXAIMYGDVSSD3Z5XUUN7SB6WAADA, seg_pos={0 0}, upload=24452]: Uploading piece...
May 26 03:37:44.801075256 (25h33m30.584415356s) [noise=true , address=145.79.227.12:4769 , piece_id=4CMAB7IQW3DY72X4R2M4D3GEDZXHQHWYEVBN3VKXMSSYABFOMJWA, seg_pos={0 0}, upload=24452]: Uploading piece...
May 26 03:37:44.802249427 (25h33m30.585589526s) [noise=true , address=145.79.227.21:4473 , piece_id=CV3E5SXKUA3OAW2QRQHHNDSK74MYME6AUNE7YD2YQJMP6MAVSQEQ, seg_pos={0 0}, upload=24452]: Uploading piece...
May 26 03:37:44.803442773 (25h33m30.586782872s) [noise=true , address=145.79.227.49:4340 , piece_id=H4PP2BCFBCRUPINAFKPEIS7IKRCDVILC4EPYEEWNTBSHTNICWGGA, seg_pos={0 0}, upload=24452]: Uploading piece...
May 26 03:37:44.804531642 (25h33m30.587871741s) [noise=true , address=77.90.31.17:4773 , piece_id=45PHIJTVNU5KTCWK4RUCJ3MF2TJSA4B37W4CG6UJ3GFRRPQQ5WZA, seg_pos={0 0}, upload=24452]: Uploading piece...
May 26 03:37:44.805927776 (25h33m30.589267875s) [noise=true , address=77.90.31.12:4589 , piece_id=P6TPHQWR7IOYUKZ4VZKC6CH77YAJJUU2A33WTECMQPUVI2ZCUQYA, seg_pos={0 0}, upload=24452]: Uploading piece...
May 26 03:37:46.520818854 (25h33m32.304158953s) [upload=24452]: Next returned nil segment
May 26 03:37:46.520830196 (25h33m32.304170295s) [upload=24452]: Waiting for error group managing segments...
May 26 03:37:46.802301060 (25h33m32.585641159s) [noise=true , address=145.79.227.8:4332 , piece_id=LRJG7OY6SCUWAEATGPICV3SQLID25AAPL5DJHMRX4T66MOVT4ARQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil>
May 26 03:37:46.805062139 (25h33m32.588402238s) [noise=true , address=145.79.227.12:4769 , piece_id=4CMAB7IQW3DY72X4R2M4D3GEDZXHQHWYEVBN3VKXMSSYABFOMJWA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil>
May 26 03:37:46.818069497 (25h33m32.601409586s) [noise=true , address=145.79.227.49:4340 , piece_id=H4PP2BCFBCRUPINAFKPEIS7IKRCDVILC4EPYEEWNTBSHTNICWGGA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil>
May 26 03:37:46.818763521 (25h33m32.60210362s) [noise=true , address=145.79.227.26:4657 , piece_id=JCI2K722O5F2HGZR4POW3UMUCGPXQUHMSZFGJL7IL2TSY3T4LKNA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil>
May 26 03:37:46.832663342 (25h33m32.616003451s) [noise=true , address=145.79.227.21:4473 , piece_id=CV3E5SXKUA3OAW2QRQHHNDSK74MYME6AUNE7YD2YQJMP6MAVSQEQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil>
May 26 03:37:46.841016390 (25h33m32.624356489s) [noise=true , address=145.79.227.45:4234 , piece_id=2YBD3GONCSIP3A63LSVRI647YC3NTWSXUCBY5ZWEIWMT5B7D7YLQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil>
May 26 03:37:46.842493007 (25h33m32.625833106s) [noise=true , address=145.79.227.40:4049 , piece_id=BAAQV3TB3GMDZNLG6UUEMPNJ2R5E5RVSEFBTOPDQHNUXJFB6XOLA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil>
May 26 03:37:46.855901301 (25h33m32.63924139s) [noise=true , address=145.79.227.5:4224 , piece_id=QYELKAAUIN72JYBUBRWGKGOF5D4FZ6M4CV2LIRWYMX6Q3FU2TECA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil>
May 26 03:37:46.872794464 (25h33m32.656134563s) [noise=true , address=145.79.227.10:4401 , piece_id=FDK7SAUVXCQM2XHWJQMLHMLEVYVYH3FWGR4CI5KLFPPFZJRCNJKQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil>
May 26 03:37:46.904359019 (25h33m32.687699148s) [noise=true , address=145.79.227.15:4873 , piece_id=MCX4ZRVNDVT53WE7W5X7JZ53ZFVOJIUWMZBPHCXAZBUXS4RULCTA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil>
May 26 03:37:47.599588789 (25h33m33.382928888s) [noise=true , address=45.91.116.12:4227 , piece_id=VUZQ6SGI6MGKMHZOT4KZP2Z2CTUGHTQ6373WQRZPTN2KANTEGDTA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil>
May 26 03:37:47.616657837 (25h33m33.399997936s) [noise=true , address=45.91.116.4:4947 , piece_id=D2SJUTVUCPKJ2UL2UVZPCGITIXAIMYGDVSSD3Z5XUUN7SB6WAADA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil>
May 26 03:38:13.890981361 (25h33m59.67432147s) [noise=true , address=77.90.31.12:4589 , piece_id=P6TPHQWR7IOYUKZ4VZKC6CH77YAJJUU2A33WTECMQPUVI2ZCUQYA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: rpc: dial tcp 77.90.31.12:4589: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
May 26 03:38:13.891042047 (25h33m59.674382146s) [noise=true , address=77.90.31.16:4735 , piece_id=EHMPUNQ3X4KQ3OS3Y2ZSAN6QLRGPS2W6DL635MO7IN6FO7QX7LFQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: rpc: dial tcp 77.90.31.16:4735: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
May 26 03:38:13.891153340 (25h33m59.674493439s) [noise=true , address=77.90.31.17:4773 , piece_id=45PHIJTVNU5KTCWK4RUCJ3MF2TJSA4B37W4CG6UJ3GFRRPQQ5WZA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: ecclient: upload failed (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, address:77.90.31.17:4773): protocol: context deadline exceeded; context deadline exceeded
May 26 03:38:13.891203274 (25h33m59.674543393s) [noise=true , address=77.90.31.11:4558 , piece_id=SGKGDSAPYDNZ2KDMXJXCQTENMFPAGI6VAXZY3WQUBIZSRFZHPWZQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: ecclient: upload failed (node:1mGEP1n1icC2gEkyRVy6BeM4vsrgqYWaQeB5rUtxjrWQjyFhTS, address:77.90.31.11:4558): protocol: write tcp 188.166.225.170:40140->77.90.31.11:4558: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
May 26 03:38:13.891231057 (25h33m59.674571156s) [seg_pos={0 0}, upload=24452]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
May 26 03:38:14.130774583 (25h33m59.914114683s) [upload=24452]: Done uploading segments...
May 26 03:38:14.130801014 (25h33m59.914141113s) [upload=24452]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 5.736085173s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [...................X.....................................X..] duration:
4.769758485s trace:
https://jaeger.dev.storj.io/trace/2f0b656030b9f519
FAIL (2026-05-26 04:23:39 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 4.769787761s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)antiblip:
4 out of 60 is failed (threshold=3) [.........X............X..........X................X.........] duration:
6.24642297s trace:
https://jaeger.dev.storj.io/trace/20840979a8f7088f
FAIL (2026-05-26 04:22:58 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
25083]: Uploading piece...
May 26 04:17:28.318973560 (26h13m14.10231366s) [noise=true , address=212.102.52.217:4420 , piece_id=PQM3W3VAO5T2BBPQEREGSHGZ3EYBGJ7LFORGCKUY6NTUP2NNE5PA, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.319026442 (26h13m14.102366541s) [noise=true , address=212.102.52.216:4397 , piece_id=5UNX4TPU63Q4AG726PSLVBFWX7ZZJL766ULEHEIIU3S4LG3EUGDA, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.319245079 (26h13m14.102585178s) [noise=true , address=212.102.52.213:4343 , piece_id=AUT3LEUM7MPZIL33QGT47V7QOOZTHR3A2W2S6EMKJBY4B3AUA4ZA, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.319586399 (26h13m14.102926499s) [noise=true , address=212.102.52.216:4402 , piece_id=VNTORDVWCQPSO5C2QUQUI76CBCTFB7TIQSEL7MWCCYRJ22T4WIBA, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.319879910 (26h13m14.103220009s) [noise=true , address=212.102.52.217:4436 , piece_id=SWQCVL47FBVYZFZIQVYNAD4NYDQLBBJL3S2EYVNHGS2Y54RRAN6A, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.320133124 (26h13m14.103473223s) [noise=true , address=212.102.52.215:4379 , piece_id=FBM4P5FPAP6OCVSHW5QYJOJWRXCO63HY2I6K3EYZKL6BRIUMME5Q, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.320357862 (26h13m14.103697961s) [noise=true , address=212.102.52.214:4362 , piece_id=LB7JGSWGNGU7BQX4FZQLVA2IUDQJT7ZP5PFXOQV4NLYXH7JAFRHA, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.320571960 (26h13m14.103912059s) [noise=true , address=212.102.52.216:4408 , piece_id=23VTNWQ4FKNZ4O552CZQERQ5KMMHSCP4Y46PNSMWU4HGHSDFLIOA, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.320588152 (26h13m14.103928251s) [noise=true , address=212.102.52.213:4344 , piece_id=UEGMZ7I63LXBAKRWSFQDHBZBXPCHG2NMZSGW4PFVHFQORZLRMQYA, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.321070803 (26h13m14.104410912s) [noise=true , address=212.102.52.217:4433 , piece_id=FGN6M3NBSA3ZOLHYPT3THZV4THHNFYSI2R6BDB4AWSQZRCZ2M7TQ, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.321436691 (26h13m14.10477679s) [noise=true , address=212.102.52.215:4380 , piece_id=6XJ45LCPGAIOCFHI35GODZRUNRS2Z63RF6WIKFUXV5R3GMSH7YJQ, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.321852404 (26h13m14.105192503s) [noise=true , address=212.102.52.213:4333 , piece_id=O4MDU2BWFAOHKWNQ3KS24KNT6VYDQ4MGLBGCWW4WZZTPMVBT4OCQ, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.322253419 (26h13m14.105593518s) [noise=true , address=212.102.52.214:4347 , piece_id=VPYOK5SVHEPZ36OOB7PYDAAINGIOUACK4FVDZBMJLWKMWC2D4NXQ, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.322652791 (26h13m14.1059929s) [noise=true , address=212.102.52.213:4331 , piece_id=LTU5YKIGVWFRQXGCECWGW656T6JMUDM7RRK6QD5CRGBB5IKMZXTQ, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.324215282 (26h13m14.107555381s) [noise=true , address=212.102.52.214:4349 , piece_id=HH47IVRCTHTDW4RHMJTEXGNB4ZK62NU6LDP5AS4FVXWPFINGHFNQ, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:29.390980339 (26h13m15.174320438s) [upload=25083]: Next returned nil segment
May 26 04:17:29.390993564 (26h13m15.174333663s) [upload=25083]: Waiting for error group managing segments...
May 26 04:17:29.673854379 (26h13m15.457194479s) [noise=true , address=212.102.52.215:4381 , piece_id=5UZ45SYYXIOMUYBHLRRP6NECZCQ5JW2RIK5MQIV2LMYQBVOON4ZA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.674402245 (26h13m15.457742344s) [noise=true , address=212.102.52.217:4433 , piece_id=FGN6M3NBSA3ZOLHYPT3THZV4THHNFYSI2R6BDB4AWSQZRCZ2M7TQ, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.677827470 (26h13m15.461167569s) [noise=true , address=212.102.52.213:4344 , piece_id=UEGMZ7I63LXBAKRWSFQDHBZBXPCHG2NMZSGW4PFVHFQORZLRMQYA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.678368041 (26h13m15.46170814s) [noise=true , address=212.102.52.215:4379 , piece_id=FBM4P5FPAP6OCVSHW5QYJOJWRXCO63HY2I6K3EYZKL6BRIUMME5Q, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.680215035 (26h13m15.463555134s) [noise=true , address=212.102.52.216:4402 , piece_id=VNTORDVWCQPSO5C2QUQUI76CBCTFB7TIQSEL7MWCCYRJ22T4WIBA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.682230351 (26h13m15.46557045s) [noise=true , address=212.102.52.217:4420 , piece_id=PQM3W3VAO5T2BBPQEREGSHGZ3EYBGJ7LFORGCKUY6NTUP2NNE5PA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.685255182 (26h13m15.468595281s) [noise=true , address=212.102.52.216:4405 , piece_id=65IGD4H7EMWEDYW2TFH643JJ5QW2BKHVFZA6D77MCHRYA6T6ESGQ, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.686472445 (26h13m15.469812544s) [noise=true , address=212.102.52.214:4362 , piece_id=LB7JGSWGNGU7BQX4FZQLVA2IUDQJT7ZP5PFXOQV4NLYXH7JAFRHA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.689410080 (26h13m15.472750179s) [noise=true , address=212.102.52.217:4436 , piece_id=SWQCVL47FBVYZFZIQVYNAD4NYDQLBBJL3S2EYVNHGS2Y54RRAN6A, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.689953157 (26h13m15.473293256s) [noise=true , address=212.102.52.214:4355 , piece_id=RKSUNVZS66X4A7ZY7WEUQXSWOTK2H5VAR7TBDICVC7MGVHA7LTGA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.694371046 (26h13m15.477711155s) [noise=true , address=212.102.52.214:4347 , piece_id=VPYOK5SVHEPZ36OOB7PYDAAINGIOUACK4FVDZBMJLWKMWC2D4NXQ, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.698718109 (26h13m15.482058208s) [noise=true , address=212.102.52.214:4349 , piece_id=HH47IVRCTHTDW4RHMJTEXGNB4ZK62NU6LDP5AS4FVXWPFINGHFNQ, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.699290191 (26h13m15.48263029s) [noise=true , address=212.102.52.215:4389 , piece_id=2OTWYJIV6A2646HAW5TWBTSQVT25V2PBKVANLGGK5SWNVHE3WGWA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.701256312 (26h13m15.484596411s) [noise=true , address=212.102.52.216:4397 , piece_id=5UNX4TPU63Q4AG726PSLVBFWX7ZZJL766ULEHEIIU3S4LG3EUGDA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.735255313 (26h13m15.518595412s) [noise=true , address=212.102.52.213:4331 , piece_id=LTU5YKIGVWFRQXGCECWGW656T6JMUDM7RRK6QD5CRGBB5IKMZXTQ, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.738998335 (26h13m15.522338434s) [noise=true , address=212.102.52.215:4380 , piece_id=6XJ45LCPGAIOCFHI35GODZRUNRS2Z63RF6WIKFUXV5R3GMSH7YJQ, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.739008105 (26h13m15.522348204s) [seg_pos={0 0}, upload=25083]: Satisfied cohort requirements after 16 pieces
May 26 04:17:29.739201553 (26h13m15.522541652s) [noise=true , address=212.102.52.217:4423 , piece_id=2KWQV3P3UYSFMBJ5BJNIHGVNZZXSCD3DEJXUV7SYVDALJSXIJAPA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TgGtX5XRdWH1qn6V5fZpQ8cY5oA2NyZx1AM17XANmr99AG9oh): protocol: expected piece hash; context canceled; EOF
May 26 04:17:29.739208446 (26h13m15.522548545s) [noise=true , address=212.102.52.217:4423 , piece_id=2KWQV3P3UYSFMBJ5BJNIHGVNZZXSCD3DEJXUV7SYVDALJSXIJAPA, seg_pos={0 0}, upload=25083]: Long tail cancellation due to optimal threshold
May 26 04:17:29.739277749 (26h13m15.522617858s) [noise=true , address=212.102.52.213:4343 , piece_id=AUT3LEUM7MPZIL33QGT47V7QOOZTHR3A2W2S6EMKJBY4B3AUA4ZA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129SbAN2aTSXzo8JLELV2ssdkmT5k7kbTcvxSZE4sFsVaKw1Aok): protocol: expected piece hash; context canceled; EOF
May 26 04:17:29.739288459 (26h13m15.522628598s) [noise=true , address=212.102.52.213:4343 , piece_id=AUT3LEUM7MPZIL33QGT47V7QOOZTHR3A2W2S6EMKJBY4B3AUA4ZA, seg_pos={0 0}, upload=25083]: Long tail cancellation due to optimal threshold
May 26 04:17:29.739324538 (26h13m15.522664637s) [noise=true , address=212.102.52.216:4408 , piece_id=23VTNWQ4FKNZ4O552CZQERQ5KMMHSCP4Y46PNSMWU4HGHSDFLIOA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BsdeZy1odVCAgcjxU6KdveXhUUTJKviRrrHLNCUuVergW6zVk): protocol: expected piece hash; context canceled; EOF
May 26 04:17:29.739330078 (26h13m15.522670177s) [noise=true , address=212.102.52.216:4408 , piece_id=23VTNWQ4FKNZ4O552CZQERQ5KMMHSCP4Y46PNSMWU4HGHSDFLIOA, seg_pos={0 0}, upload=25083]: Long tail cancellation due to optimal threshold
May 26 04:17:29.739420791 (26h13m15.52276089s) [noise=true , address=212.102.52.213:4333 , piece_id=O4MDU2BWFAOHKWNQ3KS24KNT6VYDQ4MGLBGCWW4WZZTPMVBT4OCQ, seg_pos={0 0}, upload=25083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Gm3eZxZPxPeHqLQ32kpXMpCgnwRoP8epBYtiFF1EH3U9LEazY): protocol: expected piece hash; context canceled; EOF
May 26 04:17:29.739426021 (26h13m15.52276612s) [noise=true , address=212.102.52.213:4333 , piece_id=O4MDU2BWFAOHKWNQ3KS24KNT6VYDQ4MGLBGCWW4WZZTPMVBT4OCQ, seg_pos={0 0}, upload=25083]: Long tail cancellation due to optimal threshold
May 26 04:17:29.739475536 (26h13m15.522815635s) [seg_pos={0 0}, upload=25083]: Done waiting for segment. successful: 16 optimal: 16 errs: <nil>
May 26 04:17:29.739535340 (26h13m15.522875439s) [upload=25083]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 26 04:17:29.739552654 (26h13m15.522892753s) [upload=25083]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 26 04:17:39.024892360 (26h13m24.808232459s) [upload=25083]: Done uploading segments...
May 26 04:17:39.024959289 (26h13m24.808299388s) [upload=25083]: Upload finished. err: metaclient: context deadline exceeded
Duration: 6.246447286s
core
uplink
uk-select-1
uk1-placement-17
|
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)antiblip:
4 out of 60 is failed (threshold=3) [..........X............X..........X................X........] duration:
26.571741137s trace:
https://jaeger.dev.storj.io/trace/20840979a8f7088f
FAIL (2026-05-26 04:22:40 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
25083]: Uploading piece...
May 26 04:17:28.318973560 (26h13m14.10231366s) [noise=true , address=212.102.52.217:4420 , piece_id=PQM3W3VAO5T2BBPQEREGSHGZ3EYBGJ7LFORGCKUY6NTUP2NNE5PA, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.319026442 (26h13m14.102366541s) [noise=true , address=212.102.52.216:4397 , piece_id=5UNX4TPU63Q4AG726PSLVBFWX7ZZJL766ULEHEIIU3S4LG3EUGDA, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.319245079 (26h13m14.102585178s) [noise=true , address=212.102.52.213:4343 , piece_id=AUT3LEUM7MPZIL33QGT47V7QOOZTHR3A2W2S6EMKJBY4B3AUA4ZA, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.319586399 (26h13m14.102926499s) [noise=true , address=212.102.52.216:4402 , piece_id=VNTORDVWCQPSO5C2QUQUI76CBCTFB7TIQSEL7MWCCYRJ22T4WIBA, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.319879910 (26h13m14.103220009s) [noise=true , address=212.102.52.217:4436 , piece_id=SWQCVL47FBVYZFZIQVYNAD4NYDQLBBJL3S2EYVNHGS2Y54RRAN6A, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.320133124 (26h13m14.103473223s) [noise=true , address=212.102.52.215:4379 , piece_id=FBM4P5FPAP6OCVSHW5QYJOJWRXCO63HY2I6K3EYZKL6BRIUMME5Q, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.320357862 (26h13m14.103697961s) [noise=true , address=212.102.52.214:4362 , piece_id=LB7JGSWGNGU7BQX4FZQLVA2IUDQJT7ZP5PFXOQV4NLYXH7JAFRHA, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.320571960 (26h13m14.103912059s) [noise=true , address=212.102.52.216:4408 , piece_id=23VTNWQ4FKNZ4O552CZQERQ5KMMHSCP4Y46PNSMWU4HGHSDFLIOA, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.320588152 (26h13m14.103928251s) [noise=true , address=212.102.52.213:4344 , piece_id=UEGMZ7I63LXBAKRWSFQDHBZBXPCHG2NMZSGW4PFVHFQORZLRMQYA, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.321070803 (26h13m14.104410912s) [noise=true , address=212.102.52.217:4433 , piece_id=FGN6M3NBSA3ZOLHYPT3THZV4THHNFYSI2R6BDB4AWSQZRCZ2M7TQ, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.321436691 (26h13m14.10477679s) [noise=true , address=212.102.52.215:4380 , piece_id=6XJ45LCPGAIOCFHI35GODZRUNRS2Z63RF6WIKFUXV5R3GMSH7YJQ, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.321852404 (26h13m14.105192503s) [noise=true , address=212.102.52.213:4333 , piece_id=O4MDU2BWFAOHKWNQ3KS24KNT6VYDQ4MGLBGCWW4WZZTPMVBT4OCQ, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.322253419 (26h13m14.105593518s) [noise=true , address=212.102.52.214:4347 , piece_id=VPYOK5SVHEPZ36OOB7PYDAAINGIOUACK4FVDZBMJLWKMWC2D4NXQ, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.322652791 (26h13m14.1059929s) [noise=true , address=212.102.52.213:4331 , piece_id=LTU5YKIGVWFRQXGCECWGW656T6JMUDM7RRK6QD5CRGBB5IKMZXTQ, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.324215282 (26h13m14.107555381s) [noise=true , address=212.102.52.214:4349 , piece_id=HH47IVRCTHTDW4RHMJTEXGNB4ZK62NU6LDP5AS4FVXWPFINGHFNQ, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:29.390980339 (26h13m15.174320438s) [upload=25083]: Next returned nil segment
May 26 04:17:29.390993564 (26h13m15.174333663s) [upload=25083]: Waiting for error group managing segments...
May 26 04:17:29.673854379 (26h13m15.457194479s) [noise=true , address=212.102.52.215:4381 , piece_id=5UZ45SYYXIOMUYBHLRRP6NECZCQ5JW2RIK5MQIV2LMYQBVOON4ZA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.674402245 (26h13m15.457742344s) [noise=true , address=212.102.52.217:4433 , piece_id=FGN6M3NBSA3ZOLHYPT3THZV4THHNFYSI2R6BDB4AWSQZRCZ2M7TQ, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.677827470 (26h13m15.461167569s) [noise=true , address=212.102.52.213:4344 , piece_id=UEGMZ7I63LXBAKRWSFQDHBZBXPCHG2NMZSGW4PFVHFQORZLRMQYA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.678368041 (26h13m15.46170814s) [noise=true , address=212.102.52.215:4379 , piece_id=FBM4P5FPAP6OCVSHW5QYJOJWRXCO63HY2I6K3EYZKL6BRIUMME5Q, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.680215035 (26h13m15.463555134s) [noise=true , address=212.102.52.216:4402 , piece_id=VNTORDVWCQPSO5C2QUQUI76CBCTFB7TIQSEL7MWCCYRJ22T4WIBA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.682230351 (26h13m15.46557045s) [noise=true , address=212.102.52.217:4420 , piece_id=PQM3W3VAO5T2BBPQEREGSHGZ3EYBGJ7LFORGCKUY6NTUP2NNE5PA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.685255182 (26h13m15.468595281s) [noise=true , address=212.102.52.216:4405 , piece_id=65IGD4H7EMWEDYW2TFH643JJ5QW2BKHVFZA6D77MCHRYA6T6ESGQ, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.686472445 (26h13m15.469812544s) [noise=true , address=212.102.52.214:4362 , piece_id=LB7JGSWGNGU7BQX4FZQLVA2IUDQJT7ZP5PFXOQV4NLYXH7JAFRHA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.689410080 (26h13m15.472750179s) [noise=true , address=212.102.52.217:4436 , piece_id=SWQCVL47FBVYZFZIQVYNAD4NYDQLBBJL3S2EYVNHGS2Y54RRAN6A, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.689953157 (26h13m15.473293256s) [noise=true , address=212.102.52.214:4355 , piece_id=RKSUNVZS66X4A7ZY7WEUQXSWOTK2H5VAR7TBDICVC7MGVHA7LTGA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.694371046 (26h13m15.477711155s) [noise=true , address=212.102.52.214:4347 , piece_id=VPYOK5SVHEPZ36OOB7PYDAAINGIOUACK4FVDZBMJLWKMWC2D4NXQ, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.698718109 (26h13m15.482058208s) [noise=true , address=212.102.52.214:4349 , piece_id=HH47IVRCTHTDW4RHMJTEXGNB4ZK62NU6LDP5AS4FVXWPFINGHFNQ, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.699290191 (26h13m15.48263029s) [noise=true , address=212.102.52.215:4389 , piece_id=2OTWYJIV6A2646HAW5TWBTSQVT25V2PBKVANLGGK5SWNVHE3WGWA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.701256312 (26h13m15.484596411s) [noise=true , address=212.102.52.216:4397 , piece_id=5UNX4TPU63Q4AG726PSLVBFWX7ZZJL766ULEHEIIU3S4LG3EUGDA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.735255313 (26h13m15.518595412s) [noise=true , address=212.102.52.213:4331 , piece_id=LTU5YKIGVWFRQXGCECWGW656T6JMUDM7RRK6QD5CRGBB5IKMZXTQ, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.738998335 (26h13m15.522338434s) [noise=true , address=212.102.52.215:4380 , piece_id=6XJ45LCPGAIOCFHI35GODZRUNRS2Z63RF6WIKFUXV5R3GMSH7YJQ, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.739008105 (26h13m15.522348204s) [seg_pos={0 0}, upload=25083]: Satisfied cohort requirements after 16 pieces
May 26 04:17:29.739201553 (26h13m15.522541652s) [noise=true , address=212.102.52.217:4423 , piece_id=2KWQV3P3UYSFMBJ5BJNIHGVNZZXSCD3DEJXUV7SYVDALJSXIJAPA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TgGtX5XRdWH1qn6V5fZpQ8cY5oA2NyZx1AM17XANmr99AG9oh): protocol: expected piece hash; context canceled; EOF
May 26 04:17:29.739208446 (26h13m15.522548545s) [noise=true , address=212.102.52.217:4423 , piece_id=2KWQV3P3UYSFMBJ5BJNIHGVNZZXSCD3DEJXUV7SYVDALJSXIJAPA, seg_pos={0 0}, upload=25083]: Long tail cancellation due to optimal threshold
May 26 04:17:29.739277749 (26h13m15.522617858s) [noise=true , address=212.102.52.213:4343 , piece_id=AUT3LEUM7MPZIL33QGT47V7QOOZTHR3A2W2S6EMKJBY4B3AUA4ZA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129SbAN2aTSXzo8JLELV2ssdkmT5k7kbTcvxSZE4sFsVaKw1Aok): protocol: expected piece hash; context canceled; EOF
May 26 04:17:29.739288459 (26h13m15.522628598s) [noise=true , address=212.102.52.213:4343 , piece_id=AUT3LEUM7MPZIL33QGT47V7QOOZTHR3A2W2S6EMKJBY4B3AUA4ZA, seg_pos={0 0}, upload=25083]: Long tail cancellation due to optimal threshold
May 26 04:17:29.739324538 (26h13m15.522664637s) [noise=true , address=212.102.52.216:4408 , piece_id=23VTNWQ4FKNZ4O552CZQERQ5KMMHSCP4Y46PNSMWU4HGHSDFLIOA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BsdeZy1odVCAgcjxU6KdveXhUUTJKviRrrHLNCUuVergW6zVk): protocol: expected piece hash; context canceled; EOF
May 26 04:17:29.739330078 (26h13m15.522670177s) [noise=true , address=212.102.52.216:4408 , piece_id=23VTNWQ4FKNZ4O552CZQERQ5KMMHSCP4Y46PNSMWU4HGHSDFLIOA, seg_pos={0 0}, upload=25083]: Long tail cancellation due to optimal threshold
May 26 04:17:29.739420791 (26h13m15.52276089s) [noise=true , address=212.102.52.213:4333 , piece_id=O4MDU2BWFAOHKWNQ3KS24KNT6VYDQ4MGLBGCWW4WZZTPMVBT4OCQ, seg_pos={0 0}, upload=25083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Gm3eZxZPxPeHqLQ32kpXMpCgnwRoP8epBYtiFF1EH3U9LEazY): protocol: expected piece hash; context canceled; EOF
May 26 04:17:29.739426021 (26h13m15.52276612s) [noise=true , address=212.102.52.213:4333 , piece_id=O4MDU2BWFAOHKWNQ3KS24KNT6VYDQ4MGLBGCWW4WZZTPMVBT4OCQ, seg_pos={0 0}, upload=25083]: Long tail cancellation due to optimal threshold
May 26 04:17:29.739475536 (26h13m15.522815635s) [seg_pos={0 0}, upload=25083]: Done waiting for segment. successful: 16 optimal: 16 errs: <nil>
May 26 04:17:29.739535340 (26h13m15.522875439s) [upload=25083]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 26 04:17:29.739552654 (26h13m15.522892753s) [upload=25083]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 26 04:17:39.024892360 (26h13m24.808232459s) [upload=25083]: Done uploading segments...
May 26 04:17:39.024959289 (26h13m24.808299388s) [upload=25083]: Upload finished. err: metaclient: context deadline exceeded
Duration: 26.571759882s
core
uplink
uk-select-1
uk1-placement-17
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [....X.......................................................] duration:
9.033945741s trace:
https://jaeger.dev.storj.io/trace/2f99fbcd176b1617
FAIL (2026-05-26 04:22:35 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
May 26 03:37:44.554844381 (25h33m30.33818448s) [upload=24452]: Starting upload
May 26 03:37:44.554914775 (25h33m30.338254884s) [upload=24452]: Done starting upload
May 26 03:37:44.555010087 (25h33m30.338350186s) [upload=24452]: Uploading segments...
May 26 03:37:44.555154713 (25h33m30.338494812s) [upload=24452]: Got next segment. Inline: false
May 26 03:37:44.555190892 (25h33m30.338530991s) [upload=24452]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
May 26 03:37:44.555194178 (25h33m30.338534277s) [upload=24452]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
May 26 03:37:44.796393493 (25h33m30.579733593s) [seg_pos={0 0}, upload=24452]: Begin upload segment...
May 26 03:37:44.796434813 (25h33m30.579774902s) [seg_pos={0 0}, upload=24452]: Done begin upload segment.
May 26 03:37:44.796455061 (25h33m30.57979516s) [noise=true , address=145.79.227.26:4657 , piece_id=JCI2K722O5F2HGZR4POW3UMUCGPXQUHMSZFGJL7IL2TSY3T4LKNA, seg_pos={0 0}, upload=24452]: Uploading piece...
May 26 03:37:44.796479167 (25h33m30.579819266s) [noise=true , address=77.90.31.11:4558 , piece_id=SGKGDSAPYDNZ2KDMXJXCQTENMFPAGI6VAXZY3WQUBIZSRFZHPWZQ, seg_pos={0 0}, upload=24452]: Uploading piece...
May 26 03:37:44.796547607 (25h33m30.579887706s) [noise=true , address=145.79.227.15:4873 , piece_id=MCX4ZRVNDVT53WE7W5X7JZ53ZFVOJIUWMZBPHCXAZBUXS4RULCTA, seg_pos={0 0}, upload=24452]: Uploading piece...
May 26 03:37:44.796944846 (25h33m30.580284935s) [noise=true , address=145.79.227.5:4224 , piece_id=QYELKAAUIN72JYBUBRWGKGOF5D4FZ6M4CV2LIRWYMX6Q3FU2TECA, seg_pos={0 0}, upload=24452]: Uploading piece...
May 26 03:37:44.797283402 (25h33m30.580623501s) [noise=true , address=77.90.31.16:4735 , piece_id=EHMPUNQ3X4KQ3OS3Y2ZSAN6QLRGPS2W6DL635MO7IN6FO7QX7LFQ, seg_pos={0 0}, upload=24452]: Uploading piece...
May 26 03:37:44.797625474 (25h33m30.580965573s) [noise=true , address=145.79.227.40:4049 , piece_id=BAAQV3TB3GMDZNLG6UUEMPNJ2R5E5RVSEFBTOPDQHNUXJFB6XOLA, seg_pos={0 0}, upload=24452]: Uploading piece...
May 26 03:37:44.798009868 (25h33m30.581349967s) [noise=true , address=145.79.227.8:4332 , piece_id=LRJG7OY6SCUWAEATGPICV3SQLID25AAPL5DJHMRX4T66MOVT4ARQ, seg_pos={0 0}, upload=24452]: Uploading piece...
May 26 03:37:44.798332895 (25h33m30.581672994s) [noise=true , address=145.79.227.45:4234 , piece_id=2YBD3GONCSIP3A63LSVRI647YC3NTWSXUCBY5ZWEIWMT5B7D7YLQ, seg_pos={0 0}, upload=24452]: Uploading piece...
May 26 03:37:44.798639480 (25h33m30.581979569s) [noise=true , address=145.79.227.10:4401 , piece_id=FDK7SAUVXCQM2XHWJQMLHMLEVYVYH3FWGR4CI5KLFPPFZJRCNJKQ, seg_pos={0 0}, upload=24452]: Uploading piece...
May 26 03:37:44.799006550 (25h33m30.582346649s) [noise=true , address=45.91.116.12:4227 , piece_id=VUZQ6SGI6MGKMHZOT4KZP2Z2CTUGHTQ6373WQRZPTN2KANTEGDTA, seg_pos={0 0}, upload=24452]: Uploading piece...
May 26 03:37:44.799936012 (25h33m30.583276102s) [noise=true , address=45.91.116.4:4947 , piece_id=D2SJUTVUCPKJ2UL2UVZPCGITIXAIMYGDVSSD3Z5XUUN7SB6WAADA, seg_pos={0 0}, upload=24452]: Uploading piece...
May 26 03:37:44.801075256 (25h33m30.584415356s) [noise=true , address=145.79.227.12:4769 , piece_id=4CMAB7IQW3DY72X4R2M4D3GEDZXHQHWYEVBN3VKXMSSYABFOMJWA, seg_pos={0 0}, upload=24452]: Uploading piece...
May 26 03:37:44.802249427 (25h33m30.585589526s) [noise=true , address=145.79.227.21:4473 , piece_id=CV3E5SXKUA3OAW2QRQHHNDSK74MYME6AUNE7YD2YQJMP6MAVSQEQ, seg_pos={0 0}, upload=24452]: Uploading piece...
May 26 03:37:44.803442773 (25h33m30.586782872s) [noise=true , address=145.79.227.49:4340 , piece_id=H4PP2BCFBCRUPINAFKPEIS7IKRCDVILC4EPYEEWNTBSHTNICWGGA, seg_pos={0 0}, upload=24452]: Uploading piece...
May 26 03:37:44.804531642 (25h33m30.587871741s) [noise=true , address=77.90.31.17:4773 , piece_id=45PHIJTVNU5KTCWK4RUCJ3MF2TJSA4B37W4CG6UJ3GFRRPQQ5WZA, seg_pos={0 0}, upload=24452]: Uploading piece...
May 26 03:37:44.805927776 (25h33m30.589267875s) [noise=true , address=77.90.31.12:4589 , piece_id=P6TPHQWR7IOYUKZ4VZKC6CH77YAJJUU2A33WTECMQPUVI2ZCUQYA, seg_pos={0 0}, upload=24452]: Uploading piece...
May 26 03:37:46.520818854 (25h33m32.304158953s) [upload=24452]: Next returned nil segment
May 26 03:37:46.520830196 (25h33m32.304170295s) [upload=24452]: Waiting for error group managing segments...
May 26 03:37:46.802301060 (25h33m32.585641159s) [noise=true , address=145.79.227.8:4332 , piece_id=LRJG7OY6SCUWAEATGPICV3SQLID25AAPL5DJHMRX4T66MOVT4ARQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil>
May 26 03:37:46.805062139 (25h33m32.588402238s) [noise=true , address=145.79.227.12:4769 , piece_id=4CMAB7IQW3DY72X4R2M4D3GEDZXHQHWYEVBN3VKXMSSYABFOMJWA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil>
May 26 03:37:46.818069497 (25h33m32.601409586s) [noise=true , address=145.79.227.49:4340 , piece_id=H4PP2BCFBCRUPINAFKPEIS7IKRCDVILC4EPYEEWNTBSHTNICWGGA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil>
May 26 03:37:46.818763521 (25h33m32.60210362s) [noise=true , address=145.79.227.26:4657 , piece_id=JCI2K722O5F2HGZR4POW3UMUCGPXQUHMSZFGJL7IL2TSY3T4LKNA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil>
May 26 03:37:46.832663342 (25h33m32.616003451s) [noise=true , address=145.79.227.21:4473 , piece_id=CV3E5SXKUA3OAW2QRQHHNDSK74MYME6AUNE7YD2YQJMP6MAVSQEQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil>
May 26 03:37:46.841016390 (25h33m32.624356489s) [noise=true , address=145.79.227.45:4234 , piece_id=2YBD3GONCSIP3A63LSVRI647YC3NTWSXUCBY5ZWEIWMT5B7D7YLQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil>
May 26 03:37:46.842493007 (25h33m32.625833106s) [noise=true , address=145.79.227.40:4049 , piece_id=BAAQV3TB3GMDZNLG6UUEMPNJ2R5E5RVSEFBTOPDQHNUXJFB6XOLA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil>
May 26 03:37:46.855901301 (25h33m32.63924139s) [noise=true , address=145.79.227.5:4224 , piece_id=QYELKAAUIN72JYBUBRWGKGOF5D4FZ6M4CV2LIRWYMX6Q3FU2TECA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil>
May 26 03:37:46.872794464 (25h33m32.656134563s) [noise=true , address=145.79.227.10:4401 , piece_id=FDK7SAUVXCQM2XHWJQMLHMLEVYVYH3FWGR4CI5KLFPPFZJRCNJKQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil>
May 26 03:37:46.904359019 (25h33m32.687699148s) [noise=true , address=145.79.227.15:4873 , piece_id=MCX4ZRVNDVT53WE7W5X7JZ53ZFVOJIUWMZBPHCXAZBUXS4RULCTA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil>
May 26 03:37:47.599588789 (25h33m33.382928888s) [noise=true , address=45.91.116.12:4227 , piece_id=VUZQ6SGI6MGKMHZOT4KZP2Z2CTUGHTQ6373WQRZPTN2KANTEGDTA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil>
May 26 03:37:47.616657837 (25h33m33.399997936s) [noise=true , address=45.91.116.4:4947 , piece_id=D2SJUTVUCPKJ2UL2UVZPCGITIXAIMYGDVSSD3Z5XUUN7SB6WAADA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil>
May 26 03:38:13.890981361 (25h33m59.67432147s) [noise=true , address=77.90.31.12:4589 , piece_id=P6TPHQWR7IOYUKZ4VZKC6CH77YAJJUU2A33WTECMQPUVI2ZCUQYA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: rpc: dial tcp 77.90.31.12:4589: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
May 26 03:38:13.891042047 (25h33m59.674382146s) [noise=true , address=77.90.31.16:4735 , piece_id=EHMPUNQ3X4KQ3OS3Y2ZSAN6QLRGPS2W6DL635MO7IN6FO7QX7LFQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: rpc: dial tcp 77.90.31.16:4735: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
May 26 03:38:13.891153340 (25h33m59.674493439s) [noise=true , address=77.90.31.17:4773 , piece_id=45PHIJTVNU5KTCWK4RUCJ3MF2TJSA4B37W4CG6UJ3GFRRPQQ5WZA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: ecclient: upload failed (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, address:77.90.31.17:4773): protocol: context deadline exceeded; context deadline exceeded
May 26 03:38:13.891203274 (25h33m59.674543393s) [noise=true , address=77.90.31.11:4558 , piece_id=SGKGDSAPYDNZ2KDMXJXCQTENMFPAGI6VAXZY3WQUBIZSRFZHPWZQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: ecclient: upload failed (node:1mGEP1n1icC2gEkyRVy6BeM4vsrgqYWaQeB5rUtxjrWQjyFhTS, address:77.90.31.11:4558): protocol: write tcp 188.166.225.170:40140->77.90.31.11:4558: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
May 26 03:38:13.891231057 (25h33m59.674571156s) [seg_pos={0 0}, upload=24452]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
May 26 03:38:14.130774583 (25h33m59.914114683s) [upload=24452]: Done uploading segments...
May 26 03:38:14.130801014 (25h33m59.914141113s) [upload=24452]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 9.033969627s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [....................X.....................................X.] duration:
3.86048846s trace:
https://jaeger.dev.storj.io/trace/2f0b656030b9f519
FAIL (2026-05-26 04:22:28 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.86050934s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)antiblip:
4 out of 60 is failed (threshold=3) [...........X............X..........X................X.......] duration:
4.116856034s trace:
https://jaeger.dev.storj.io/trace/20840979a8f7088f
FAIL (2026-05-26 04:22:01 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
25083]: Uploading piece...
May 26 04:17:28.318973560 (26h13m14.10231366s) [noise=true , address=212.102.52.217:4420 , piece_id=PQM3W3VAO5T2BBPQEREGSHGZ3EYBGJ7LFORGCKUY6NTUP2NNE5PA, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.319026442 (26h13m14.102366541s) [noise=true , address=212.102.52.216:4397 , piece_id=5UNX4TPU63Q4AG726PSLVBFWX7ZZJL766ULEHEIIU3S4LG3EUGDA, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.319245079 (26h13m14.102585178s) [noise=true , address=212.102.52.213:4343 , piece_id=AUT3LEUM7MPZIL33QGT47V7QOOZTHR3A2W2S6EMKJBY4B3AUA4ZA, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.319586399 (26h13m14.102926499s) [noise=true , address=212.102.52.216:4402 , piece_id=VNTORDVWCQPSO5C2QUQUI76CBCTFB7TIQSEL7MWCCYRJ22T4WIBA, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.319879910 (26h13m14.103220009s) [noise=true , address=212.102.52.217:4436 , piece_id=SWQCVL47FBVYZFZIQVYNAD4NYDQLBBJL3S2EYVNHGS2Y54RRAN6A, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.320133124 (26h13m14.103473223s) [noise=true , address=212.102.52.215:4379 , piece_id=FBM4P5FPAP6OCVSHW5QYJOJWRXCO63HY2I6K3EYZKL6BRIUMME5Q, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.320357862 (26h13m14.103697961s) [noise=true , address=212.102.52.214:4362 , piece_id=LB7JGSWGNGU7BQX4FZQLVA2IUDQJT7ZP5PFXOQV4NLYXH7JAFRHA, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.320571960 (26h13m14.103912059s) [noise=true , address=212.102.52.216:4408 , piece_id=23VTNWQ4FKNZ4O552CZQERQ5KMMHSCP4Y46PNSMWU4HGHSDFLIOA, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.320588152 (26h13m14.103928251s) [noise=true , address=212.102.52.213:4344 , piece_id=UEGMZ7I63LXBAKRWSFQDHBZBXPCHG2NMZSGW4PFVHFQORZLRMQYA, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.321070803 (26h13m14.104410912s) [noise=true , address=212.102.52.217:4433 , piece_id=FGN6M3NBSA3ZOLHYPT3THZV4THHNFYSI2R6BDB4AWSQZRCZ2M7TQ, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.321436691 (26h13m14.10477679s) [noise=true , address=212.102.52.215:4380 , piece_id=6XJ45LCPGAIOCFHI35GODZRUNRS2Z63RF6WIKFUXV5R3GMSH7YJQ, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.321852404 (26h13m14.105192503s) [noise=true , address=212.102.52.213:4333 , piece_id=O4MDU2BWFAOHKWNQ3KS24KNT6VYDQ4MGLBGCWW4WZZTPMVBT4OCQ, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.322253419 (26h13m14.105593518s) [noise=true , address=212.102.52.214:4347 , piece_id=VPYOK5SVHEPZ36OOB7PYDAAINGIOUACK4FVDZBMJLWKMWC2D4NXQ, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.322652791 (26h13m14.1059929s) [noise=true , address=212.102.52.213:4331 , piece_id=LTU5YKIGVWFRQXGCECWGW656T6JMUDM7RRK6QD5CRGBB5IKMZXTQ, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.324215282 (26h13m14.107555381s) [noise=true , address=212.102.52.214:4349 , piece_id=HH47IVRCTHTDW4RHMJTEXGNB4ZK62NU6LDP5AS4FVXWPFINGHFNQ, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:29.390980339 (26h13m15.174320438s) [upload=25083]: Next returned nil segment
May 26 04:17:29.390993564 (26h13m15.174333663s) [upload=25083]: Waiting for error group managing segments...
May 26 04:17:29.673854379 (26h13m15.457194479s) [noise=true , address=212.102.52.215:4381 , piece_id=5UZ45SYYXIOMUYBHLRRP6NECZCQ5JW2RIK5MQIV2LMYQBVOON4ZA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.674402245 (26h13m15.457742344s) [noise=true , address=212.102.52.217:4433 , piece_id=FGN6M3NBSA3ZOLHYPT3THZV4THHNFYSI2R6BDB4AWSQZRCZ2M7TQ, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.677827470 (26h13m15.461167569s) [noise=true , address=212.102.52.213:4344 , piece_id=UEGMZ7I63LXBAKRWSFQDHBZBXPCHG2NMZSGW4PFVHFQORZLRMQYA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.678368041 (26h13m15.46170814s) [noise=true , address=212.102.52.215:4379 , piece_id=FBM4P5FPAP6OCVSHW5QYJOJWRXCO63HY2I6K3EYZKL6BRIUMME5Q, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.680215035 (26h13m15.463555134s) [noise=true , address=212.102.52.216:4402 , piece_id=VNTORDVWCQPSO5C2QUQUI76CBCTFB7TIQSEL7MWCCYRJ22T4WIBA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.682230351 (26h13m15.46557045s) [noise=true , address=212.102.52.217:4420 , piece_id=PQM3W3VAO5T2BBPQEREGSHGZ3EYBGJ7LFORGCKUY6NTUP2NNE5PA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.685255182 (26h13m15.468595281s) [noise=true , address=212.102.52.216:4405 , piece_id=65IGD4H7EMWEDYW2TFH643JJ5QW2BKHVFZA6D77MCHRYA6T6ESGQ, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.686472445 (26h13m15.469812544s) [noise=true , address=212.102.52.214:4362 , piece_id=LB7JGSWGNGU7BQX4FZQLVA2IUDQJT7ZP5PFXOQV4NLYXH7JAFRHA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.689410080 (26h13m15.472750179s) [noise=true , address=212.102.52.217:4436 , piece_id=SWQCVL47FBVYZFZIQVYNAD4NYDQLBBJL3S2EYVNHGS2Y54RRAN6A, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.689953157 (26h13m15.473293256s) [noise=true , address=212.102.52.214:4355 , piece_id=RKSUNVZS66X4A7ZY7WEUQXSWOTK2H5VAR7TBDICVC7MGVHA7LTGA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.694371046 (26h13m15.477711155s) [noise=true , address=212.102.52.214:4347 , piece_id=VPYOK5SVHEPZ36OOB7PYDAAINGIOUACK4FVDZBMJLWKMWC2D4NXQ, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.698718109 (26h13m15.482058208s) [noise=true , address=212.102.52.214:4349 , piece_id=HH47IVRCTHTDW4RHMJTEXGNB4ZK62NU6LDP5AS4FVXWPFINGHFNQ, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.699290191 (26h13m15.48263029s) [noise=true , address=212.102.52.215:4389 , piece_id=2OTWYJIV6A2646HAW5TWBTSQVT25V2PBKVANLGGK5SWNVHE3WGWA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.701256312 (26h13m15.484596411s) [noise=true , address=212.102.52.216:4397 , piece_id=5UNX4TPU63Q4AG726PSLVBFWX7ZZJL766ULEHEIIU3S4LG3EUGDA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.735255313 (26h13m15.518595412s) [noise=true , address=212.102.52.213:4331 , piece_id=LTU5YKIGVWFRQXGCECWGW656T6JMUDM7RRK6QD5CRGBB5IKMZXTQ, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.738998335 (26h13m15.522338434s) [noise=true , address=212.102.52.215:4380 , piece_id=6XJ45LCPGAIOCFHI35GODZRUNRS2Z63RF6WIKFUXV5R3GMSH7YJQ, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.739008105 (26h13m15.522348204s) [seg_pos={0 0}, upload=25083]: Satisfied cohort requirements after 16 pieces
May 26 04:17:29.739201553 (26h13m15.522541652s) [noise=true , address=212.102.52.217:4423 , piece_id=2KWQV3P3UYSFMBJ5BJNIHGVNZZXSCD3DEJXUV7SYVDALJSXIJAPA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TgGtX5XRdWH1qn6V5fZpQ8cY5oA2NyZx1AM17XANmr99AG9oh): protocol: expected piece hash; context canceled; EOF
May 26 04:17:29.739208446 (26h13m15.522548545s) [noise=true , address=212.102.52.217:4423 , piece_id=2KWQV3P3UYSFMBJ5BJNIHGVNZZXSCD3DEJXUV7SYVDALJSXIJAPA, seg_pos={0 0}, upload=25083]: Long tail cancellation due to optimal threshold
May 26 04:17:29.739277749 (26h13m15.522617858s) [noise=true , address=212.102.52.213:4343 , piece_id=AUT3LEUM7MPZIL33QGT47V7QOOZTHR3A2W2S6EMKJBY4B3AUA4ZA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129SbAN2aTSXzo8JLELV2ssdkmT5k7kbTcvxSZE4sFsVaKw1Aok): protocol: expected piece hash; context canceled; EOF
May 26 04:17:29.739288459 (26h13m15.522628598s) [noise=true , address=212.102.52.213:4343 , piece_id=AUT3LEUM7MPZIL33QGT47V7QOOZTHR3A2W2S6EMKJBY4B3AUA4ZA, seg_pos={0 0}, upload=25083]: Long tail cancellation due to optimal threshold
May 26 04:17:29.739324538 (26h13m15.522664637s) [noise=true , address=212.102.52.216:4408 , piece_id=23VTNWQ4FKNZ4O552CZQERQ5KMMHSCP4Y46PNSMWU4HGHSDFLIOA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BsdeZy1odVCAgcjxU6KdveXhUUTJKviRrrHLNCUuVergW6zVk): protocol: expected piece hash; context canceled; EOF
May 26 04:17:29.739330078 (26h13m15.522670177s) [noise=true , address=212.102.52.216:4408 , piece_id=23VTNWQ4FKNZ4O552CZQERQ5KMMHSCP4Y46PNSMWU4HGHSDFLIOA, seg_pos={0 0}, upload=25083]: Long tail cancellation due to optimal threshold
May 26 04:17:29.739420791 (26h13m15.52276089s) [noise=true , address=212.102.52.213:4333 , piece_id=O4MDU2BWFAOHKWNQ3KS24KNT6VYDQ4MGLBGCWW4WZZTPMVBT4OCQ, seg_pos={0 0}, upload=25083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Gm3eZxZPxPeHqLQ32kpXMpCgnwRoP8epBYtiFF1EH3U9LEazY): protocol: expected piece hash; context canceled; EOF
May 26 04:17:29.739426021 (26h13m15.52276612s) [noise=true , address=212.102.52.213:4333 , piece_id=O4MDU2BWFAOHKWNQ3KS24KNT6VYDQ4MGLBGCWW4WZZTPMVBT4OCQ, seg_pos={0 0}, upload=25083]: Long tail cancellation due to optimal threshold
May 26 04:17:29.739475536 (26h13m15.522815635s) [seg_pos={0 0}, upload=25083]: Done waiting for segment. successful: 16 optimal: 16 errs: <nil>
May 26 04:17:29.739535340 (26h13m15.522875439s) [upload=25083]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 26 04:17:29.739552654 (26h13m15.522892753s) [upload=25083]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 26 04:17:39.024892360 (26h13m24.808232459s) [upload=25083]: Done uploading segments...
May 26 04:17:39.024959289 (26h13m24.808299388s) [upload=25083]: Upload finished. err: metaclient: context deadline exceeded
Duration: 4.116880541s
core
uplink
uk-select-1
uk1-placement-17
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [.....................X.....................................X] duration:
5.000601466s trace:
https://jaeger.dev.storj.io/trace/2f0b656030b9f519
FAIL (2026-05-26 04:21:48 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.000625893s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.....X......................................................] duration:
5.808522592s trace:
https://jaeger.dev.storj.io/trace/2f99fbcd176b1617
FAIL (2026-05-26 04:21:35 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
May 26 03:37:44.554844381 (25h33m30.33818448s) [upload=24452]: Starting upload
May 26 03:37:44.554914775 (25h33m30.338254884s) [upload=24452]: Done starting upload
May 26 03:37:44.555010087 (25h33m30.338350186s) [upload=24452]: Uploading segments...
May 26 03:37:44.555154713 (25h33m30.338494812s) [upload=24452]: Got next segment. Inline: false
May 26 03:37:44.555190892 (25h33m30.338530991s) [upload=24452]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
May 26 03:37:44.555194178 (25h33m30.338534277s) [upload=24452]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
May 26 03:37:44.796393493 (25h33m30.579733593s) [seg_pos={0 0}, upload=24452]: Begin upload segment...
May 26 03:37:44.796434813 (25h33m30.579774902s) [seg_pos={0 0}, upload=24452]: Done begin upload segment.
May 26 03:37:44.796455061 (25h33m30.57979516s) [noise=true , address=145.79.227.26:4657 , piece_id=JCI2K722O5F2HGZR4POW3UMUCGPXQUHMSZFGJL7IL2TSY3T4LKNA, seg_pos={0 0}, upload=24452]: Uploading piece...
May 26 03:37:44.796479167 (25h33m30.579819266s) [noise=true , address=77.90.31.11:4558 , piece_id=SGKGDSAPYDNZ2KDMXJXCQTENMFPAGI6VAXZY3WQUBIZSRFZHPWZQ, seg_pos={0 0}, upload=24452]: Uploading piece...
May 26 03:37:44.796547607 (25h33m30.579887706s) [noise=true , address=145.79.227.15:4873 , piece_id=MCX4ZRVNDVT53WE7W5X7JZ53ZFVOJIUWMZBPHCXAZBUXS4RULCTA, seg_pos={0 0}, upload=24452]: Uploading piece...
May 26 03:37:44.796944846 (25h33m30.580284935s) [noise=true , address=145.79.227.5:4224 , piece_id=QYELKAAUIN72JYBUBRWGKGOF5D4FZ6M4CV2LIRWYMX6Q3FU2TECA, seg_pos={0 0}, upload=24452]: Uploading piece...
May 26 03:37:44.797283402 (25h33m30.580623501s) [noise=true , address=77.90.31.16:4735 , piece_id=EHMPUNQ3X4KQ3OS3Y2ZSAN6QLRGPS2W6DL635MO7IN6FO7QX7LFQ, seg_pos={0 0}, upload=24452]: Uploading piece...
May 26 03:37:44.797625474 (25h33m30.580965573s) [noise=true , address=145.79.227.40:4049 , piece_id=BAAQV3TB3GMDZNLG6UUEMPNJ2R5E5RVSEFBTOPDQHNUXJFB6XOLA, seg_pos={0 0}, upload=24452]: Uploading piece...
May 26 03:37:44.798009868 (25h33m30.581349967s) [noise=true , address=145.79.227.8:4332 , piece_id=LRJG7OY6SCUWAEATGPICV3SQLID25AAPL5DJHMRX4T66MOVT4ARQ, seg_pos={0 0}, upload=24452]: Uploading piece...
May 26 03:37:44.798332895 (25h33m30.581672994s) [noise=true , address=145.79.227.45:4234 , piece_id=2YBD3GONCSIP3A63LSVRI647YC3NTWSXUCBY5ZWEIWMT5B7D7YLQ, seg_pos={0 0}, upload=24452]: Uploading piece...
May 26 03:37:44.798639480 (25h33m30.581979569s) [noise=true , address=145.79.227.10:4401 , piece_id=FDK7SAUVXCQM2XHWJQMLHMLEVYVYH3FWGR4CI5KLFPPFZJRCNJKQ, seg_pos={0 0}, upload=24452]: Uploading piece...
May 26 03:37:44.799006550 (25h33m30.582346649s) [noise=true , address=45.91.116.12:4227 , piece_id=VUZQ6SGI6MGKMHZOT4KZP2Z2CTUGHTQ6373WQRZPTN2KANTEGDTA, seg_pos={0 0}, upload=24452]: Uploading piece...
May 26 03:37:44.799936012 (25h33m30.583276102s) [noise=true , address=45.91.116.4:4947 , piece_id=D2SJUTVUCPKJ2UL2UVZPCGITIXAIMYGDVSSD3Z5XUUN7SB6WAADA, seg_pos={0 0}, upload=24452]: Uploading piece...
May 26 03:37:44.801075256 (25h33m30.584415356s) [noise=true , address=145.79.227.12:4769 , piece_id=4CMAB7IQW3DY72X4R2M4D3GEDZXHQHWYEVBN3VKXMSSYABFOMJWA, seg_pos={0 0}, upload=24452]: Uploading piece...
May 26 03:37:44.802249427 (25h33m30.585589526s) [noise=true , address=145.79.227.21:4473 , piece_id=CV3E5SXKUA3OAW2QRQHHNDSK74MYME6AUNE7YD2YQJMP6MAVSQEQ, seg_pos={0 0}, upload=24452]: Uploading piece...
May 26 03:37:44.803442773 (25h33m30.586782872s) [noise=true , address=145.79.227.49:4340 , piece_id=H4PP2BCFBCRUPINAFKPEIS7IKRCDVILC4EPYEEWNTBSHTNICWGGA, seg_pos={0 0}, upload=24452]: Uploading piece...
May 26 03:37:44.804531642 (25h33m30.587871741s) [noise=true , address=77.90.31.17:4773 , piece_id=45PHIJTVNU5KTCWK4RUCJ3MF2TJSA4B37W4CG6UJ3GFRRPQQ5WZA, seg_pos={0 0}, upload=24452]: Uploading piece...
May 26 03:37:44.805927776 (25h33m30.589267875s) [noise=true , address=77.90.31.12:4589 , piece_id=P6TPHQWR7IOYUKZ4VZKC6CH77YAJJUU2A33WTECMQPUVI2ZCUQYA, seg_pos={0 0}, upload=24452]: Uploading piece...
May 26 03:37:46.520818854 (25h33m32.304158953s) [upload=24452]: Next returned nil segment
May 26 03:37:46.520830196 (25h33m32.304170295s) [upload=24452]: Waiting for error group managing segments...
May 26 03:37:46.802301060 (25h33m32.585641159s) [noise=true , address=145.79.227.8:4332 , piece_id=LRJG7OY6SCUWAEATGPICV3SQLID25AAPL5DJHMRX4T66MOVT4ARQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil>
May 26 03:37:46.805062139 (25h33m32.588402238s) [noise=true , address=145.79.227.12:4769 , piece_id=4CMAB7IQW3DY72X4R2M4D3GEDZXHQHWYEVBN3VKXMSSYABFOMJWA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil>
May 26 03:37:46.818069497 (25h33m32.601409586s) [noise=true , address=145.79.227.49:4340 , piece_id=H4PP2BCFBCRUPINAFKPEIS7IKRCDVILC4EPYEEWNTBSHTNICWGGA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil>
May 26 03:37:46.818763521 (25h33m32.60210362s) [noise=true , address=145.79.227.26:4657 , piece_id=JCI2K722O5F2HGZR4POW3UMUCGPXQUHMSZFGJL7IL2TSY3T4LKNA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil>
May 26 03:37:46.832663342 (25h33m32.616003451s) [noise=true , address=145.79.227.21:4473 , piece_id=CV3E5SXKUA3OAW2QRQHHNDSK74MYME6AUNE7YD2YQJMP6MAVSQEQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil>
May 26 03:37:46.841016390 (25h33m32.624356489s) [noise=true , address=145.79.227.45:4234 , piece_id=2YBD3GONCSIP3A63LSVRI647YC3NTWSXUCBY5ZWEIWMT5B7D7YLQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil>
May 26 03:37:46.842493007 (25h33m32.625833106s) [noise=true , address=145.79.227.40:4049 , piece_id=BAAQV3TB3GMDZNLG6UUEMPNJ2R5E5RVSEFBTOPDQHNUXJFB6XOLA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil>
May 26 03:37:46.855901301 (25h33m32.63924139s) [noise=true , address=145.79.227.5:4224 , piece_id=QYELKAAUIN72JYBUBRWGKGOF5D4FZ6M4CV2LIRWYMX6Q3FU2TECA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil>
May 26 03:37:46.872794464 (25h33m32.656134563s) [noise=true , address=145.79.227.10:4401 , piece_id=FDK7SAUVXCQM2XHWJQMLHMLEVYVYH3FWGR4CI5KLFPPFZJRCNJKQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil>
May 26 03:37:46.904359019 (25h33m32.687699148s) [noise=true , address=145.79.227.15:4873 , piece_id=MCX4ZRVNDVT53WE7W5X7JZ53ZFVOJIUWMZBPHCXAZBUXS4RULCTA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil>
May 26 03:37:47.599588789 (25h33m33.382928888s) [noise=true , address=45.91.116.12:4227 , piece_id=VUZQ6SGI6MGKMHZOT4KZP2Z2CTUGHTQ6373WQRZPTN2KANTEGDTA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil>
May 26 03:37:47.616657837 (25h33m33.399997936s) [noise=true , address=45.91.116.4:4947 , piece_id=D2SJUTVUCPKJ2UL2UVZPCGITIXAIMYGDVSSD3Z5XUUN7SB6WAADA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil>
May 26 03:38:13.890981361 (25h33m59.67432147s) [noise=true , address=77.90.31.12:4589 , piece_id=P6TPHQWR7IOYUKZ4VZKC6CH77YAJJUU2A33WTECMQPUVI2ZCUQYA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: rpc: dial tcp 77.90.31.12:4589: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
May 26 03:38:13.891042047 (25h33m59.674382146s) [noise=true , address=77.90.31.16:4735 , piece_id=EHMPUNQ3X4KQ3OS3Y2ZSAN6QLRGPS2W6DL635MO7IN6FO7QX7LFQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: rpc: dial tcp 77.90.31.16:4735: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
May 26 03:38:13.891153340 (25h33m59.674493439s) [noise=true , address=77.90.31.17:4773 , piece_id=45PHIJTVNU5KTCWK4RUCJ3MF2TJSA4B37W4CG6UJ3GFRRPQQ5WZA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: ecclient: upload failed (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, address:77.90.31.17:4773): protocol: context deadline exceeded; context deadline exceeded
May 26 03:38:13.891203274 (25h33m59.674543393s) [noise=true , address=77.90.31.11:4558 , piece_id=SGKGDSAPYDNZ2KDMXJXCQTENMFPAGI6VAXZY3WQUBIZSRFZHPWZQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: ecclient: upload failed (node:1mGEP1n1icC2gEkyRVy6BeM4vsrgqYWaQeB5rUtxjrWQjyFhTS, address:77.90.31.11:4558): protocol: write tcp 188.166.225.170:40140->77.90.31.11:4558: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
May 26 03:38:13.891231057 (25h33m59.674571156s) [seg_pos={0 0}, upload=24452]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
May 26 03:38:14.130774583 (25h33m59.914114683s) [upload=24452]: Done uploading segments...
May 26 03:38:14.130801014 (25h33m59.914141113s) [upload=24452]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 5.808543543s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [......................X.....................................] duration:
3.731505845s trace:
https://jaeger.dev.storj.io/trace/5e3081a3454ea1e7
FAIL (2026-05-26 04:21:26 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.731523339s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)antiblip:
4 out of 60 is failed (threshold=3) [............X............X..........X................X......] duration:
3.948490111s trace:
https://jaeger.dev.storj.io/trace/20840979a8f7088f
FAIL (2026-05-26 04:20:58 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
25083]: Uploading piece...
May 26 04:17:28.318973560 (26h13m14.10231366s) [noise=true , address=212.102.52.217:4420 , piece_id=PQM3W3VAO5T2BBPQEREGSHGZ3EYBGJ7LFORGCKUY6NTUP2NNE5PA, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.319026442 (26h13m14.102366541s) [noise=true , address=212.102.52.216:4397 , piece_id=5UNX4TPU63Q4AG726PSLVBFWX7ZZJL766ULEHEIIU3S4LG3EUGDA, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.319245079 (26h13m14.102585178s) [noise=true , address=212.102.52.213:4343 , piece_id=AUT3LEUM7MPZIL33QGT47V7QOOZTHR3A2W2S6EMKJBY4B3AUA4ZA, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.319586399 (26h13m14.102926499s) [noise=true , address=212.102.52.216:4402 , piece_id=VNTORDVWCQPSO5C2QUQUI76CBCTFB7TIQSEL7MWCCYRJ22T4WIBA, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.319879910 (26h13m14.103220009s) [noise=true , address=212.102.52.217:4436 , piece_id=SWQCVL47FBVYZFZIQVYNAD4NYDQLBBJL3S2EYVNHGS2Y54RRAN6A, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.320133124 (26h13m14.103473223s) [noise=true , address=212.102.52.215:4379 , piece_id=FBM4P5FPAP6OCVSHW5QYJOJWRXCO63HY2I6K3EYZKL6BRIUMME5Q, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.320357862 (26h13m14.103697961s) [noise=true , address=212.102.52.214:4362 , piece_id=LB7JGSWGNGU7BQX4FZQLVA2IUDQJT7ZP5PFXOQV4NLYXH7JAFRHA, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.320571960 (26h13m14.103912059s) [noise=true , address=212.102.52.216:4408 , piece_id=23VTNWQ4FKNZ4O552CZQERQ5KMMHSCP4Y46PNSMWU4HGHSDFLIOA, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.320588152 (26h13m14.103928251s) [noise=true , address=212.102.52.213:4344 , piece_id=UEGMZ7I63LXBAKRWSFQDHBZBXPCHG2NMZSGW4PFVHFQORZLRMQYA, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.321070803 (26h13m14.104410912s) [noise=true , address=212.102.52.217:4433 , piece_id=FGN6M3NBSA3ZOLHYPT3THZV4THHNFYSI2R6BDB4AWSQZRCZ2M7TQ, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.321436691 (26h13m14.10477679s) [noise=true , address=212.102.52.215:4380 , piece_id=6XJ45LCPGAIOCFHI35GODZRUNRS2Z63RF6WIKFUXV5R3GMSH7YJQ, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.321852404 (26h13m14.105192503s) [noise=true , address=212.102.52.213:4333 , piece_id=O4MDU2BWFAOHKWNQ3KS24KNT6VYDQ4MGLBGCWW4WZZTPMVBT4OCQ, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.322253419 (26h13m14.105593518s) [noise=true , address=212.102.52.214:4347 , piece_id=VPYOK5SVHEPZ36OOB7PYDAAINGIOUACK4FVDZBMJLWKMWC2D4NXQ, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.322652791 (26h13m14.1059929s) [noise=true , address=212.102.52.213:4331 , piece_id=LTU5YKIGVWFRQXGCECWGW656T6JMUDM7RRK6QD5CRGBB5IKMZXTQ, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:28.324215282 (26h13m14.107555381s) [noise=true , address=212.102.52.214:4349 , piece_id=HH47IVRCTHTDW4RHMJTEXGNB4ZK62NU6LDP5AS4FVXWPFINGHFNQ, seg_pos={0 0}, upload=25083]: Uploading piece...
May 26 04:17:29.390980339 (26h13m15.174320438s) [upload=25083]: Next returned nil segment
May 26 04:17:29.390993564 (26h13m15.174333663s) [upload=25083]: Waiting for error group managing segments...
May 26 04:17:29.673854379 (26h13m15.457194479s) [noise=true , address=212.102.52.215:4381 , piece_id=5UZ45SYYXIOMUYBHLRRP6NECZCQ5JW2RIK5MQIV2LMYQBVOON4ZA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.674402245 (26h13m15.457742344s) [noise=true , address=212.102.52.217:4433 , piece_id=FGN6M3NBSA3ZOLHYPT3THZV4THHNFYSI2R6BDB4AWSQZRCZ2M7TQ, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.677827470 (26h13m15.461167569s) [noise=true , address=212.102.52.213:4344 , piece_id=UEGMZ7I63LXBAKRWSFQDHBZBXPCHG2NMZSGW4PFVHFQORZLRMQYA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.678368041 (26h13m15.46170814s) [noise=true , address=212.102.52.215:4379 , piece_id=FBM4P5FPAP6OCVSHW5QYJOJWRXCO63HY2I6K3EYZKL6BRIUMME5Q, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.680215035 (26h13m15.463555134s) [noise=true , address=212.102.52.216:4402 , piece_id=VNTORDVWCQPSO5C2QUQUI76CBCTFB7TIQSEL7MWCCYRJ22T4WIBA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.682230351 (26h13m15.46557045s) [noise=true , address=212.102.52.217:4420 , piece_id=PQM3W3VAO5T2BBPQEREGSHGZ3EYBGJ7LFORGCKUY6NTUP2NNE5PA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.685255182 (26h13m15.468595281s) [noise=true , address=212.102.52.216:4405 , piece_id=65IGD4H7EMWEDYW2TFH643JJ5QW2BKHVFZA6D77MCHRYA6T6ESGQ, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.686472445 (26h13m15.469812544s) [noise=true , address=212.102.52.214:4362 , piece_id=LB7JGSWGNGU7BQX4FZQLVA2IUDQJT7ZP5PFXOQV4NLYXH7JAFRHA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.689410080 (26h13m15.472750179s) [noise=true , address=212.102.52.217:4436 , piece_id=SWQCVL47FBVYZFZIQVYNAD4NYDQLBBJL3S2EYVNHGS2Y54RRAN6A, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.689953157 (26h13m15.473293256s) [noise=true , address=212.102.52.214:4355 , piece_id=RKSUNVZS66X4A7ZY7WEUQXSWOTK2H5VAR7TBDICVC7MGVHA7LTGA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.694371046 (26h13m15.477711155s) [noise=true , address=212.102.52.214:4347 , piece_id=VPYOK5SVHEPZ36OOB7PYDAAINGIOUACK4FVDZBMJLWKMWC2D4NXQ, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.698718109 (26h13m15.482058208s) [noise=true , address=212.102.52.214:4349 , piece_id=HH47IVRCTHTDW4RHMJTEXGNB4ZK62NU6LDP5AS4FVXWPFINGHFNQ, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.699290191 (26h13m15.48263029s) [noise=true , address=212.102.52.215:4389 , piece_id=2OTWYJIV6A2646HAW5TWBTSQVT25V2PBKVANLGGK5SWNVHE3WGWA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.701256312 (26h13m15.484596411s) [noise=true , address=212.102.52.216:4397 , piece_id=5UNX4TPU63Q4AG726PSLVBFWX7ZZJL766ULEHEIIU3S4LG3EUGDA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.735255313 (26h13m15.518595412s) [noise=true , address=212.102.52.213:4331 , piece_id=LTU5YKIGVWFRQXGCECWGW656T6JMUDM7RRK6QD5CRGBB5IKMZXTQ, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.738998335 (26h13m15.522338434s) [noise=true , address=212.102.52.215:4380 , piece_id=6XJ45LCPGAIOCFHI35GODZRUNRS2Z63RF6WIKFUXV5R3GMSH7YJQ, seg_pos={0 0}, upload=25083]: Done uploading piece. err: <nil>
May 26 04:17:29.739008105 (26h13m15.522348204s) [seg_pos={0 0}, upload=25083]: Satisfied cohort requirements after 16 pieces
May 26 04:17:29.739201553 (26h13m15.522541652s) [noise=true , address=212.102.52.217:4423 , piece_id=2KWQV3P3UYSFMBJ5BJNIHGVNZZXSCD3DEJXUV7SYVDALJSXIJAPA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TgGtX5XRdWH1qn6V5fZpQ8cY5oA2NyZx1AM17XANmr99AG9oh): protocol: expected piece hash; context canceled; EOF
May 26 04:17:29.739208446 (26h13m15.522548545s) [noise=true , address=212.102.52.217:4423 , piece_id=2KWQV3P3UYSFMBJ5BJNIHGVNZZXSCD3DEJXUV7SYVDALJSXIJAPA, seg_pos={0 0}, upload=25083]: Long tail cancellation due to optimal threshold
May 26 04:17:29.739277749 (26h13m15.522617858s) [noise=true , address=212.102.52.213:4343 , piece_id=AUT3LEUM7MPZIL33QGT47V7QOOZTHR3A2W2S6EMKJBY4B3AUA4ZA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129SbAN2aTSXzo8JLELV2ssdkmT5k7kbTcvxSZE4sFsVaKw1Aok): protocol: expected piece hash; context canceled; EOF
May 26 04:17:29.739288459 (26h13m15.522628598s) [noise=true , address=212.102.52.213:4343 , piece_id=AUT3LEUM7MPZIL33QGT47V7QOOZTHR3A2W2S6EMKJBY4B3AUA4ZA, seg_pos={0 0}, upload=25083]: Long tail cancellation due to optimal threshold
May 26 04:17:29.739324538 (26h13m15.522664637s) [noise=true , address=212.102.52.216:4408 , piece_id=23VTNWQ4FKNZ4O552CZQERQ5KMMHSCP4Y46PNSMWU4HGHSDFLIOA, seg_pos={0 0}, upload=25083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BsdeZy1odVCAgcjxU6KdveXhUUTJKviRrrHLNCUuVergW6zVk): protocol: expected piece hash; context canceled; EOF
May 26 04:17:29.739330078 (26h13m15.522670177s) [noise=true , address=212.102.52.216:4408 , piece_id=23VTNWQ4FKNZ4O552CZQERQ5KMMHSCP4Y46PNSMWU4HGHSDFLIOA, seg_pos={0 0}, upload=25083]: Long tail cancellation due to optimal threshold
May 26 04:17:29.739420791 (26h13m15.52276089s) [noise=true , address=212.102.52.213:4333 , piece_id=O4MDU2BWFAOHKWNQ3KS24KNT6VYDQ4MGLBGCWW4WZZTPMVBT4OCQ, seg_pos={0 0}, upload=25083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Gm3eZxZPxPeHqLQ32kpXMpCgnwRoP8epBYtiFF1EH3U9LEazY): protocol: expected piece hash; context canceled; EOF
May 26 04:17:29.739426021 (26h13m15.52276612s) [noise=true , address=212.102.52.213:4333 , piece_id=O4MDU2BWFAOHKWNQ3KS24KNT6VYDQ4MGLBGCWW4WZZTPMVBT4OCQ, seg_pos={0 0}, upload=25083]: Long tail cancellation due to optimal threshold
May 26 04:17:29.739475536 (26h13m15.522815635s) [seg_pos={0 0}, upload=25083]: Done waiting for segment. successful: 16 optimal: 16 errs: <nil>
May 26 04:17:29.739535340 (26h13m15.522875439s) [upload=25083]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 26 04:17:29.739552654 (26h13m15.522892753s) [upload=25083]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 26 04:17:39.024892360 (26h13m24.808232459s) [upload=25083]: Done uploading segments...
May 26 04:17:39.024959289 (26h13m24.808299388s) [upload=25083]: Upload finished. err: metaclient: context deadline exceeded
Duration: 3.948513375s
core
uplink
uk-select-1
uk1-placement-17
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [.......................X....................................] duration:
3.878274688s trace:
https://jaeger.dev.storj.io/trace/5e3081a3454ea1e7
FAIL (2026-05-26 04:20:58 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.87829687s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [........................X...................................] duration:
3.668694173s trace:
https://jaeger.dev.storj.io/trace/5e3081a3454ea1e7
FAIL (2026-05-26 04:20:40 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.668716516s
core
uplink
us-select-1
us1-placement-20
|