Orbiter History

Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure

Orbiter Check History

Showing 50 of 363549 reports (Page 252 of 7271)

checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...............................X...X.......................X] duration: 30.001836106s trace: https://jaeger.dev.storj.io/trace/170bd04d853994ae FAIL (2025-12-31 12:10:48 UTC) orbiter warning: context deadline exceeded Duration: 30.001863098s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [................X..........X.X......X.............X.........] duration: 3.337097582s trace: https://jaeger.dev.storj.io/trace/2c96062d1b5b5a44 FAIL (2025-12-31 12:10:08 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.337129331s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 35.071130109s trace: https://jaeger.dev.storj.io/trace/3696061a2156809c FAIL (2025-12-31 12:10:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== .17:4794 , piece_id=ZCNDE7ID7HX43BDZ7KFQKZ6R5V5KUPZWCIRRUQNK65I73JMZQ2JQ, seg_pos={0 0}, upload=543695]: Uploading piece... Dec 31 11:56:12.262721887 (620h27m12.133804932s) [noise=true , address=193.149.164.143:4695 , piece_id=5CJUMEUXTJVQOCSHQKZD4ZHQBTFSMUPIQCI7QMJX2URBSUAIGIFA, seg_pos={0 0}, upload=543695]: Uploading piece... Dec 31 11:56:12.263130455 (620h27m12.1342135s) [noise=true , address=45.91.116.5:4326 , piece_id=VDLNHILWQD6ZTD7JMHQY5BDH2A4IHSJTVHJGDI4IK3OBDZSLOADA, seg_pos={0 0}, upload=543695]: Uploading piece... Dec 31 11:56:12.264175605 (620h27m12.13525865s) [noise=true , address=77.90.31.18:4812 , piece_id=KOGGG55EKOFTTT7OOZ5AULZHU2POHF5DMMSX2RM5M65FC3Y3AXRQ, seg_pos={0 0}, upload=543695]: Uploading piece... Dec 31 11:56:12.264193529 (620h27m12.135276574s) [noise=true , address=145.79.227.49:4358 , piece_id=RP2Y6L6CZD3XOL7XOU3HRDSYZZ6AENXT7LM6XVHZPIZAHSL26BPQ, seg_pos={0 0}, upload=543695]: Uploading piece... Dec 31 11:56:12.264609000 (620h27m12.135692035s) [noise=true , address=45.91.116.11:4145 , piece_id=ZX5FCIOENG6476SS6RDITOCPPCDBXAYOHAOAP4Q545FEBKSSQDDA, seg_pos={0 0}, upload=543695]: Uploading piece... Dec 31 11:56:12.264666079 (620h27m12.135749114s) [noise=true , address=64.49.13.140:4505 , piece_id=YRQ2VLRASSQVB65ONBLZXVLXY7COOSYVUUNMKJO5AOAJKFFIEW2A, seg_pos={0 0}, upload=543695]: Uploading piece... Dec 31 11:56:12.265310566 (620h27m12.136393611s) [noise=true , address=145.79.227.44:4206 , piece_id=TR747XVZACWUD6WGRSROIO5OHFLVBM3XT3XTEWA74GXNXMMYMQHQ, seg_pos={0 0}, upload=543695]: Uploading piece... Dec 31 11:56:12.265745504 (620h27m12.136828549s) [noise=true , address=145.79.227.33:4894 , piece_id=TFZVFOU3GA45OWXOWVA4R2A3W47CS6DCK5K4DAMYYMIH4J5CD23A, seg_pos={0 0}, upload=543695]: Uploading piece... Dec 31 11:56:12.266321861 (620h27m12.137404896s) [noise=true , address=45.91.116.7:4013 , piece_id=A4VDKYMR74JC325EFEP22V3TOGZRMTQB4UOE5I2Y7JKQF32AIZAA, seg_pos={0 0}, upload=543695]: Uploading piece... Dec 31 11:56:12.266868933 (620h27m12.137951978s) [noise=true , address=64.49.13.169:4707 , piece_id=ZQTZNZZQFGHXU2DIVBDUMJTYRAWAPRATLOGQ6AY4A6N45276NDXQ, seg_pos={0 0}, upload=543695]: Uploading piece... Dec 31 11:56:12.267258635 (620h27m12.13834167s) [noise=true , address=77.90.31.13:4657 , piece_id=KSNYKBDG4YOLLPTKC4OFSQR6XOBOGNA64PC3BCKQLRLYGCWFCUUQ, seg_pos={0 0}, upload=543695]: Uploading piece... Dec 31 11:56:12.267587522 (620h27m12.138670567s) [noise=true , address=145.79.227.40:4058 , piece_id=F3RLQOLC5C6JRKLBZICBMGRSJENDRTTP3PGJEFAOBXNHZP4QI3IA, seg_pos={0 0}, upload=543695]: Uploading piece... Dec 31 11:56:12.268440155 (620h27m12.13952319s) [noise=true , address=145.79.227.34:4939 , piece_id=TSCIX7EBMZ72GAOPGWHMCUW4PVEONBZ56A3ODVMIMY6LZZW2XDAQ, seg_pos={0 0}, upload=543695]: Uploading piece... Dec 31 11:56:12.269297889 (620h27m12.140380924s) [noise=true , address=145.79.227.2:4134 , piece_id=HWWXLVVLSMPGUKIP5A5T5IXAB3ZCABH7OA5DIJSNDXRQU23EIFNA, seg_pos={0 0}, upload=543695]: Uploading piece... Dec 31 11:56:13.698835153 (620h27m13.569918188s) [upload=543695]: Next returned nil segment Dec 31 11:56:13.698965801 (620h27m13.570048846s) [upload=543695]: Waiting for error group managing segments... Dec 31 11:56:14.047217086 (620h27m13.918300131s) [noise=true , address=64.49.13.140:4505 , piece_id=YRQ2VLRASSQVB65ONBLZXVLXY7COOSYVUUNMKJO5AOAJKFFIEW2A, seg_pos={0 0}, upload=543695]: Done uploading piece. err: <nil> Dec 31 11:56:14.137538488 (620h27m14.008621523s) [noise=true , address=145.79.227.2:4134 , piece_id=HWWXLVVLSMPGUKIP5A5T5IXAB3ZCABH7OA5DIJSNDXRQU23EIFNA, seg_pos={0 0}, upload=543695]: Done uploading piece. err: <nil> Dec 31 11:56:14.139917508 (620h27m14.011000543s) [noise=true , address=145.79.227.46:4619 , piece_id=BV3NLVQZM457CG2ZYJYZANIDKDVYCQWC64FZGRXFFDV6FFLC2SJA, seg_pos={0 0}, upload=543695]: Done uploading piece. err: <nil> Dec 31 11:56:14.472160101 (620h27m14.343243136s) [noise=true , address=77.90.31.17:4794 , piece_id=ZCNDE7ID7HX43BDZ7KFQKZ6R5V5KUPZWCIRRUQNK65I73JMZQ2JQ, seg_pos={0 0}, upload=543695]: Done uploading piece. err: <nil> Dec 31 11:56:14.476987874 (620h27m14.348070919s) [noise=true , address=45.91.116.5:4326 , piece_id=VDLNHILWQD6ZTD7JMHQY5BDH2A4IHSJTVHJGDI4IK3OBDZSLOADA, seg_pos={0 0}, upload=543695]: Done uploading piece. err: <nil> Dec 31 11:56:14.518004714 (620h27m14.389087749s) [noise=true , address=45.91.116.7:4013 , piece_id=A4VDKYMR74JC325EFEP22V3TOGZRMTQB4UOE5I2Y7JKQF32AIZAA, seg_pos={0 0}, upload=543695]: Done uploading piece. err: <nil> Dec 31 11:56:14.540552146 (620h27m14.411635191s) [noise=true , address=77.90.31.18:4812 , piece_id=KOGGG55EKOFTTT7OOZ5AULZHU2POHF5DMMSX2RM5M65FC3Y3AXRQ, seg_pos={0 0}, upload=543695]: Done uploading piece. err: <nil> Dec 31 11:56:14.677369210 (620h27m14.548452255s) [noise=true , address=145.79.227.40:4058 , piece_id=F3RLQOLC5C6JRKLBZICBMGRSJENDRTTP3PGJEFAOBXNHZP4QI3IA, seg_pos={0 0}, upload=543695]: Done uploading piece. err: <nil> Dec 31 11:56:15.034375707 (620h27m14.905458742s) [noise=true , address=145.79.227.44:4206 , piece_id=TR747XVZACWUD6WGRSROIO5OHFLVBM3XT3XTEWA74GXNXMMYMQHQ, seg_pos={0 0}, upload=543695]: Done uploading piece. err: <nil> Dec 31 11:56:15.095619932 (620h27m14.966702967s) [noise=true , address=145.79.227.49:4358 , piece_id=RP2Y6L6CZD3XOL7XOU3HRDSYZZ6AENXT7LM6XVHZPIZAHSL26BPQ, seg_pos={0 0}, upload=543695]: Done uploading piece. err: <nil> Dec 31 11:56:15.099851950 (620h27m14.970934995s) [noise=true , address=145.79.227.43:4164 , piece_id=QQ4D2S4XAMIDBDSZNYM2G5WT4V7DR4E5WHDSYVT4SZ4YHANAY4QA, seg_pos={0 0}, upload=543695]: Done uploading piece. err: <nil> Dec 31 11:56:15.113070049 (620h27m14.984153094s) [noise=true , address=145.79.227.33:4894 , piece_id=TFZVFOU3GA45OWXOWVA4R2A3W47CS6DCK5K4DAMYYMIH4J5CD23A, seg_pos={0 0}, upload=543695]: Done uploading piece. err: <nil> Dec 31 11:56:15.127931066 (620h27m14.999014101s) [noise=true , address=145.79.227.30:4792 , piece_id=B72NRZMLRWZQRV222OBU6EPXZYYF4K6PXT6L77XZKNVZOA25NLIQ, seg_pos={0 0}, upload=543695]: Done uploading piece. err: <nil> Dec 31 11:56:15.147826669 (620h27m15.018909714s) [noise=true , address=193.149.164.143:4695 , piece_id=5CJUMEUXTJVQOCSHQKZD4ZHQBTFSMUPIQCI7QMJX2URBSUAIGIFA, seg_pos={0 0}, upload=543695]: Done uploading piece. err: <nil> Dec 31 11:56:15.147845695 (620h27m15.01892874s) [seg_pos={0 0}, upload=543695]: Segment reached optimal threshold of 14 pieces. Dec 31 11:56:15.148175553 (620h27m15.019258598s) [noise=true , address=145.79.227.34:4939 , piece_id=TSCIX7EBMZ72GAOPGWHMCUW4PVEONBZ56A3ODVMIMY6LZZW2XDAQ, seg_pos={0 0}, upload=543695]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1h7xrPMs5kHtjm7hiB4hfDaZUHFgtJNoMGqdy3HFif2Aned8eu): protocol: expected piece hash; context canceled; EOF Dec 31 11:56:15.148194439 (620h27m15.019277484s) [noise=true , address=145.79.227.34:4939 , piece_id=TSCIX7EBMZ72GAOPGWHMCUW4PVEONBZ56A3ODVMIMY6LZZW2XDAQ, seg_pos={0 0}, upload=543695]: Long tail cancellation due to optimal threshold Dec 31 11:56:15.148491545 (620h27m15.01957458s) [noise=true , address=77.90.31.13:4657 , piece_id=KSNYKBDG4YOLLPTKC4OFSQR6XOBOGNA64PC3BCKQLRLYGCWFCUUQ, seg_pos={0 0}, upload=543695]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1LLSnWuMa6W7y2jhFUqXoSPEsTyrFsKi8FQDNXF8Tv7XHbP6yt): protocol: write tcp 188.166.225.170:49614->77.90.31.13:4657: use of closed network connection; write tcp 188.166.225.170:49614->77.90.31.13:4657: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49614->77.90.31.13:4657: use of closed network connection Dec 31 11:56:15.148498648 (620h27m15.019581693s) [noise=true , address=77.90.31.13:4657 , piece_id=KSNYKBDG4YOLLPTKC4OFSQR6XOBOGNA64PC3BCKQLRLYGCWFCUUQ, seg_pos={0 0}, upload=543695]: Long tail cancellation due to optimal threshold Dec 31 11:56:15.148641420 (620h27m15.019724465s) [noise=true , address=45.91.116.11:4145 , piece_id=ZX5FCIOENG6476SS6RDITOCPPCDBXAYOHAOAP4Q545FEBKSSQDDA, seg_pos={0 0}, upload=543695]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WKV8PJxq5Kyb3ubCbrfbyYaX6fvRkPMKW2ssUteMFNHm9fP4E): protocol: expected piece hash; context canceled; EOF Dec 31 11:56:15.148647441 (620h27m15.019730476s) [noise=true , address=45.91.116.11:4145 , piece_id=ZX5FCIOENG6476SS6RDITOCPPCDBXAYOHAOAP4Q545FEBKSSQDDA, seg_pos={0 0}, upload=543695]: Long tail cancellation due to optimal threshold Dec 31 11:56:15.148750738 (620h27m15.019833773s) [noise=true , address=64.49.13.169:4707 , piece_id=ZQTZNZZQFGHXU2DIVBDUMJTYRAWAPRATLOGQ6AY4A6N45276NDXQ, seg_pos={0 0}, upload=543695]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:45002->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:45002->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45002->64.49.13.169:4707: use of closed network connection Dec 31 11:56:15.148766589 (620h27m15.019849634s) [noise=true , address=64.49.13.169:4707 , piece_id=ZQTZNZZQFGHXU2DIVBDUMJTYRAWAPRATLOGQ6AY4A6N45276NDXQ, seg_pos={0 0}, upload=543695]: Long tail cancellation due to optimal threshold Dec 31 11:56:15.148782148 (620h27m15.019865193s) [seg_pos={0 0}, upload=543695]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 11:56:15.148845759 (620h27m15.019928804s) [upload=543695]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 11:56:15.148850428 (620h27m15.019933463s) [upload=543695]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 11:56:15.442319734 (620h27m15.313402779s) [upload=543695]: Done uploading segments... Dec 31 11:56:15.442337046 (620h27m15.313420081s) [upload=543695]: Upload finished. err: <nil> Duration: 35.071159455s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................................X...X.......................] duration: 9.245780866s trace: https://jaeger.dev.storj.io/trace/2e598bd65e5b1d02 FAIL (2025-12-31 12:09:33 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 9.245808058s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [X................X..........X.X......X.............X........] duration: 5.000991062s trace: https://jaeger.dev.storj.io/trace/2c96062d1b5b5a44 FAIL (2025-12-31 12:09:32 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00101704s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [...X..X..XX....X....X.....XX.XX.X....XX.....X.....X....XX...] duration: 5.938731132s trace: https://jaeger.dev.storj.io/trace/7019fd93cdb00dc6 FAIL (2025-12-31 12:09:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, Piece ID: EDYF4XRKLTRTIW4LUHX2MKBXDHB6CYT2P7FO6ZJPM623BOFWMVYA) context deadline exceeded; piecestore: (Node ID: 1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, Piece ID: KHMSB3E6X2FYMNPZ4DLBLXI4FH3O22TICPFDWGTJAGAKKRAVHZAQ) context deadline exceeded ======LOGS====== 1 12:07:11.861573379 (620h38m11.732656424s) [upload=543853]: Starting upload Dec 31 12:07:11.861635807 (620h38m11.732718842s) [upload=543853]: Done starting upload Dec 31 12:07:11.861750156 (620h38m11.732833201s) [upload=543853]: Uploading segments... Dec 31 12:07:11.861836680 (620h38m11.732919725s) [upload=543853]: Got next segment. Inline: false Dec 31 12:07:11.861868741 (620h38m11.732951786s) [upload=543853]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 12:07:11.861874061 (620h38m11.732957106s) [upload=543853]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 12:07:12.105514432 (620h38m11.976597467s) [seg_pos={0 0}, upload=543853]: Begin upload segment... Dec 31 12:07:12.105556783 (620h38m11.976639818s) [seg_pos={0 0}, upload=543853]: Done begin upload segment. Dec 31 12:07:12.105585979 (620h38m11.976669024s) [noise=true , address=145.79.227.2:4115 , piece_id=XKLPIJXVH654UTNUMQDADXOLCNVZQT46UTGCNDYNLPMIUADUCPBA, seg_pos={0 0}, upload=543853]: Uploading piece... Dec 31 12:07:12.105610175 (620h38m11.97669325s) [noise=true , address=145.79.227.17:4971 , piece_id=PH2JN2DOEDDW7KIRMPC35ONPQFFTXBFP5YQBA3N4ISH2THYZDMIA, seg_pos={0 0}, upload=543853]: Uploading piece... Dec 31 12:07:12.106090158 (620h38m11.977173193s) [noise=true , address=145.79.227.22:4509 , piece_id=XNNKDMKGP7AD4JSVOEPBNB6WC6X23B2YLGEK6IBGMLS4BCIA43EA, seg_pos={0 0}, upload=543853]: Uploading piece... Dec 31 12:07:12.106456676 (620h38m11.977539722s) [noise=true , address=77.90.31.11:4554 , piece_id=KHMSB3E6X2FYMNPZ4DLBLXI4FH3O22TICPFDWGTJAGAKKRAVHZAQ, seg_pos={0 0}, upload=543853]: Uploading piece... Dec 31 12:07:12.106742721 (620h38m11.977825766s) [noise=true , address=77.90.31.21:4878 , piece_id=EDYF4XRKLTRTIW4LUHX2MKBXDHB6CYT2P7FO6ZJPM623BOFWMVYA, seg_pos={0 0}, upload=543853]: Uploading piece... Dec 31 12:07:12.107171507 (620h38m11.978254542s) [noise=true , address=145.79.227.50:4374 , piece_id=IPDLL47PNRLUPJIGNPVRH2KOEYALGZVGJLP2M3C4KE35OAK4DVDQ, seg_pos={0 0}, upload=543853]: Uploading piece... Dec 31 12:07:12.107477240 (620h38m11.978560275s) [noise=true , address=145.79.227.4:4187 , piece_id=2DP76MR3IBTSG56BQ2RIGPKUSOABTRXBCNKROP4IHWMSCHKTSCAQ, seg_pos={0 0}, upload=543853]: Uploading piece... Dec 31 12:07:12.107847826 (620h38m11.978930861s) [noise=true , address=145.79.227.15:4874 , piece_id=RXNAVDXFC3TJQZBAJHEJ2MYHQUOXV3NV36KREZ2S7JFYYGFZ6PRA, seg_pos={0 0}, upload=543853]: Uploading piece... Dec 31 12:07:12.108205035 (620h38m11.979288071s) [noise=true , address=145.79.227.37:4566 , piece_id=MITYMLPQAJF2JJSQ4DKDVQDD3RPP7XK6AI4LHCU54XDM5NSZMOKQ, seg_pos={0 0}, upload=543853]: Uploading piece... Dec 31 12:07:12.109451609 (620h38m11.980534654s) [noise=true , address=145.79.227.16:4909 , piece_id=SAELWRFMEC3GDG5TY7JSAVEUPFPAFADTNURLWMGXD6ZYLSTP6YEA, seg_pos={0 0}, upload=543853]: Uploading piece... Dec 31 12:07:12.111851349 (620h38m11.982934394s) [noise=true , address=145.79.227.13:4802 , piece_id=DALBEF23TMYEQWXQ55JM5J2QDO4L6JIEAFBIFWHMK5WTE7MYDFUA, seg_pos={0 0}, upload=543853]: Uploading piece... Dec 31 12:07:12.113438611 (620h38m11.984521646s) [noise=true , address=77.90.31.19:4844 , piece_id=5KPGFPID7GFTLD226A7EML6LSMMEADIJLHSBREG4T2BVTL3IYHJQ, seg_pos={0 0}, upload=543853]: Uploading piece... Dec 31 12:07:12.114694452 (620h38m11.985777487s) [noise=true , address=145.79.227.43:4159 , piece_id=FRTB2LLAAIJFE3DJMK4L4XLCAWZ2BPP23BASHBGMIAWAVMNK5TQQ, seg_pos={0 0}, upload=543853]: Uploading piece... Dec 31 12:07:12.116052859 (620h38m11.987135904s) [noise=true , address=45.91.116.10:4081 , piece_id=NWOEAIHIURDVOHQUO3XV4MEFNTXJEA4L6P7IU6EVOXOQBLNPACHQ, seg_pos={0 0}, upload=543853]: Uploading piece... Dec 31 12:07:12.116061655 (620h38m11.98714469s) [noise=true , address=145.79.227.25:4619 , piece_id=MXSXTGWAEH6PENQUUHXRZ4APEK37U7BLSIXKXJETEDEQGEJFSEKA, seg_pos={0 0}, upload=543853]: Uploading piece... Dec 31 12:07:12.117313990 (620h38m11.988397035s) [noise=true , address=45.91.116.11:4118 , piece_id=QSSUS2PZCFGXIUWMXL4ZXRVORXHN5UYA7POTFQD66DFKHGKR7CJA, seg_pos={0 0}, upload=543853]: Uploading piece... Dec 31 12:07:13.828739360 (620h38m13.699822405s) [upload=543853]: Next returned nil segment Dec 31 12:07:13.828760559 (620h38m13.699843594s) [upload=543853]: Waiting for error group managing segments... Dec 31 12:07:14.107262421 (620h38m13.978345456s) [noise=true , address=145.79.227.4:4187 , piece_id=2DP76MR3IBTSG56BQ2RIGPKUSOABTRXBCNKROP4IHWMSCHKTSCAQ, seg_pos={0 0}, upload=543853]: Done uploading piece. err: <nil> Dec 31 12:07:14.117350740 (620h38m13.988433775s) [noise=true , address=145.79.227.50:4374 , piece_id=IPDLL47PNRLUPJIGNPVRH2KOEYALGZVGJLP2M3C4KE35OAK4DVDQ, seg_pos={0 0}, upload=543853]: Done uploading piece. err: <nil> Dec 31 12:07:14.118644733 (620h38m13.989727769s) [noise=true , address=145.79.227.2:4115 , piece_id=XKLPIJXVH654UTNUMQDADXOLCNVZQT46UTGCNDYNLPMIUADUCPBA, seg_pos={0 0}, upload=543853]: Done uploading piece. err: <nil> Dec 31 12:07:14.119417244 (620h38m13.990500279s) [noise=true , address=145.79.227.25:4619 , piece_id=MXSXTGWAEH6PENQUUHXRZ4APEK37U7BLSIXKXJETEDEQGEJFSEKA, seg_pos={0 0}, upload=543853]: Done uploading piece. err: <nil> Dec 31 12:07:14.122045620 (620h38m13.993128655s) [noise=true , address=145.79.227.17:4971 , piece_id=PH2JN2DOEDDW7KIRMPC35ONPQFFTXBFP5YQBA3N4ISH2THYZDMIA, seg_pos={0 0}, upload=543853]: Done uploading piece. err: <nil> Dec 31 12:07:14.128858542 (620h38m13.999941577s) [noise=true , address=145.79.227.16:4909 , piece_id=SAELWRFMEC3GDG5TY7JSAVEUPFPAFADTNURLWMGXD6ZYLSTP6YEA, seg_pos={0 0}, upload=543853]: Done uploading piece. err: <nil> Dec 31 12:07:14.132529763 (620h38m14.003612798s) [noise=true , address=145.79.227.37:4566 , piece_id=MITYMLPQAJF2JJSQ4DKDVQDD3RPP7XK6AI4LHCU54XDM5NSZMOKQ, seg_pos={0 0}, upload=543853]: Done uploading piece. err: <nil> Dec 31 12:07:14.135029943 (620h38m14.006112988s) [noise=true , address=145.79.227.43:4159 , piece_id=FRTB2LLAAIJFE3DJMK4L4XLCAWZ2BPP23BASHBGMIAWAVMNK5TQQ, seg_pos={0 0}, upload=543853]: Done uploading piece. err: <nil> Dec 31 12:07:14.182526382 (620h38m14.053609427s) [noise=true , address=145.79.227.22:4509 , piece_id=XNNKDMKGP7AD4JSVOEPBNB6WC6X23B2YLGEK6IBGMLS4BCIA43EA, seg_pos={0 0}, upload=543853]: Done uploading piece. err: <nil> Dec 31 12:07:14.191420678 (620h38m14.062503723s) [noise=true , address=145.79.227.13:4802 , piece_id=DALBEF23TMYEQWXQ55JM5J2QDO4L6JIEAFBIFWHMK5WTE7MYDFUA, seg_pos={0 0}, upload=543853]: Done uploading piece. err: <nil> Dec 31 12:07:14.192442173 (620h38m14.063525218s) [noise=true , address=145.79.227.15:4874 , piece_id=RXNAVDXFC3TJQZBAJHEJ2MYHQUOXV3NV36KREZ2S7JFYYGFZ6PRA, seg_pos={0 0}, upload=543853]: Done uploading piece. err: <nil> Dec 31 12:07:14.290745965 (620h38m14.16182903s) [noise=true , address=77.90.31.11:4554 , piece_id=KHMSB3E6X2FYMNPZ4DLBLXI4FH3O22TICPFDWGTJAGAKKRAVHZAQ, seg_pos={0 0}, upload=543853]: Done uploading piece. err: <nil> Dec 31 12:07:14.395876316 (620h38m14.266959361s) [noise=true , address=77.90.31.21:4878 , piece_id=EDYF4XRKLTRTIW4LUHX2MKBXDHB6CYT2P7FO6ZJPM623BOFWMVYA, seg_pos={0 0}, upload=543853]: Done uploading piece. err: <nil> Dec 31 12:07:14.395903037 (620h38m14.266986083s) [seg_pos={0 0}, upload=543853]: Segment reached optimal threshold of 13 pieces. Dec 31 12:07:14.396128787 (620h38m14.267211822s) [noise=true , address=45.91.116.11:4118 , piece_id=QSSUS2PZCFGXIUWMXL4ZXRVORXHN5UYA7POTFQD66DFKHGKR7CJA, seg_pos={0 0}, upload=543853]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: expected piece hash; context canceled; EOF Dec 31 12:07:14.396139528 (620h38m14.267222563s) [noise=true , address=45.91.116.11:4118 , piece_id=QSSUS2PZCFGXIUWMXL4ZXRVORXHN5UYA7POTFQD66DFKHGKR7CJA, seg_pos={0 0}, upload=543853]: Long tail cancellation due to optimal threshold Dec 31 12:07:14.396298831 (620h38m14.267381866s) [noise=true , address=45.91.116.10:4081 , piece_id=NWOEAIHIURDVOHQUO3XV4MEFNTXJEA4L6P7IU6EVOXOQBLNPACHQ, seg_pos={0 0}, upload=543853]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: expected piece hash; context canceled; EOF Dec 31 12:07:14.396308540 (620h38m14.267391585s) [noise=true , address=45.91.116.10:4081 , piece_id=NWOEAIHIURDVOHQUO3XV4MEFNTXJEA4L6P7IU6EVOXOQBLNPACHQ, seg_pos={0 0}, upload=543853]: Long tail cancellation due to optimal threshold Dec 31 12:07:14.396476529 (620h38m14.267559564s) [noise=true , address=77.90.31.19:4844 , piece_id=5KPGFPID7GFTLD226A7EML6LSMMEADIJLHSBREG4T2BVTL3IYHJQ, seg_pos={0 0}, upload=543853]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: write tcp 188.166.225.170:37356->77.90.31.19:4844: use of closed network connection; write tcp 188.166.225.170:37356->77.90.31.19:4844: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37356->77.90.31.19:4844: use of closed network connection Dec 31 12:07:14.396490626 (620h38m14.267573692s) [noise=true , address=77.90.31.19:4844 , piece_id=5KPGFPID7GFTLD226A7EML6LSMMEADIJLHSBREG4T2BVTL3IYHJQ, seg_pos={0 0}, upload=543853]: Long tail cancellation due to optimal threshold Dec 31 12:07:14.396519712 (620h38m14.267602757s) [seg_pos={0 0}, upload=543853]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 12:07:14.396606867 (620h38m14.267689913s) [upload=543853]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 12:07:14.396613249 (620h38m14.267696294s) [upload=543853]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 12:07:14.691428722 (620h38m14.562511757s) [upload=543853]: Done uploading segments... Dec 31 12:07:14.691447397 (620h38m14.562530432s) [upload=543853]: Upload finished. err: <nil> Duration: 5.938754316s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 38.023521248s trace: https://jaeger.dev.storj.io/trace/3696061a2156809c FAIL (2025-12-31 12:09:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== .17:4794 , piece_id=ZCNDE7ID7HX43BDZ7KFQKZ6R5V5KUPZWCIRRUQNK65I73JMZQ2JQ, seg_pos={0 0}, upload=543695]: Uploading piece... Dec 31 11:56:12.262721887 (620h27m12.133804932s) [noise=true , address=193.149.164.143:4695 , piece_id=5CJUMEUXTJVQOCSHQKZD4ZHQBTFSMUPIQCI7QMJX2URBSUAIGIFA, seg_pos={0 0}, upload=543695]: Uploading piece... Dec 31 11:56:12.263130455 (620h27m12.1342135s) [noise=true , address=45.91.116.5:4326 , piece_id=VDLNHILWQD6ZTD7JMHQY5BDH2A4IHSJTVHJGDI4IK3OBDZSLOADA, seg_pos={0 0}, upload=543695]: Uploading piece... Dec 31 11:56:12.264175605 (620h27m12.13525865s) [noise=true , address=77.90.31.18:4812 , piece_id=KOGGG55EKOFTTT7OOZ5AULZHU2POHF5DMMSX2RM5M65FC3Y3AXRQ, seg_pos={0 0}, upload=543695]: Uploading piece... Dec 31 11:56:12.264193529 (620h27m12.135276574s) [noise=true , address=145.79.227.49:4358 , piece_id=RP2Y6L6CZD3XOL7XOU3HRDSYZZ6AENXT7LM6XVHZPIZAHSL26BPQ, seg_pos={0 0}, upload=543695]: Uploading piece... Dec 31 11:56:12.264609000 (620h27m12.135692035s) [noise=true , address=45.91.116.11:4145 , piece_id=ZX5FCIOENG6476SS6RDITOCPPCDBXAYOHAOAP4Q545FEBKSSQDDA, seg_pos={0 0}, upload=543695]: Uploading piece... Dec 31 11:56:12.264666079 (620h27m12.135749114s) [noise=true , address=64.49.13.140:4505 , piece_id=YRQ2VLRASSQVB65ONBLZXVLXY7COOSYVUUNMKJO5AOAJKFFIEW2A, seg_pos={0 0}, upload=543695]: Uploading piece... Dec 31 11:56:12.265310566 (620h27m12.136393611s) [noise=true , address=145.79.227.44:4206 , piece_id=TR747XVZACWUD6WGRSROIO5OHFLVBM3XT3XTEWA74GXNXMMYMQHQ, seg_pos={0 0}, upload=543695]: Uploading piece... Dec 31 11:56:12.265745504 (620h27m12.136828549s) [noise=true , address=145.79.227.33:4894 , piece_id=TFZVFOU3GA45OWXOWVA4R2A3W47CS6DCK5K4DAMYYMIH4J5CD23A, seg_pos={0 0}, upload=543695]: Uploading piece... Dec 31 11:56:12.266321861 (620h27m12.137404896s) [noise=true , address=45.91.116.7:4013 , piece_id=A4VDKYMR74JC325EFEP22V3TOGZRMTQB4UOE5I2Y7JKQF32AIZAA, seg_pos={0 0}, upload=543695]: Uploading piece... Dec 31 11:56:12.266868933 (620h27m12.137951978s) [noise=true , address=64.49.13.169:4707 , piece_id=ZQTZNZZQFGHXU2DIVBDUMJTYRAWAPRATLOGQ6AY4A6N45276NDXQ, seg_pos={0 0}, upload=543695]: Uploading piece... Dec 31 11:56:12.267258635 (620h27m12.13834167s) [noise=true , address=77.90.31.13:4657 , piece_id=KSNYKBDG4YOLLPTKC4OFSQR6XOBOGNA64PC3BCKQLRLYGCWFCUUQ, seg_pos={0 0}, upload=543695]: Uploading piece... Dec 31 11:56:12.267587522 (620h27m12.138670567s) [noise=true , address=145.79.227.40:4058 , piece_id=F3RLQOLC5C6JRKLBZICBMGRSJENDRTTP3PGJEFAOBXNHZP4QI3IA, seg_pos={0 0}, upload=543695]: Uploading piece... Dec 31 11:56:12.268440155 (620h27m12.13952319s) [noise=true , address=145.79.227.34:4939 , piece_id=TSCIX7EBMZ72GAOPGWHMCUW4PVEONBZ56A3ODVMIMY6LZZW2XDAQ, seg_pos={0 0}, upload=543695]: Uploading piece... Dec 31 11:56:12.269297889 (620h27m12.140380924s) [noise=true , address=145.79.227.2:4134 , piece_id=HWWXLVVLSMPGUKIP5A5T5IXAB3ZCABH7OA5DIJSNDXRQU23EIFNA, seg_pos={0 0}, upload=543695]: Uploading piece... Dec 31 11:56:13.698835153 (620h27m13.569918188s) [upload=543695]: Next returned nil segment Dec 31 11:56:13.698965801 (620h27m13.570048846s) [upload=543695]: Waiting for error group managing segments... Dec 31 11:56:14.047217086 (620h27m13.918300131s) [noise=true , address=64.49.13.140:4505 , piece_id=YRQ2VLRASSQVB65ONBLZXVLXY7COOSYVUUNMKJO5AOAJKFFIEW2A, seg_pos={0 0}, upload=543695]: Done uploading piece. err: <nil> Dec 31 11:56:14.137538488 (620h27m14.008621523s) [noise=true , address=145.79.227.2:4134 , piece_id=HWWXLVVLSMPGUKIP5A5T5IXAB3ZCABH7OA5DIJSNDXRQU23EIFNA, seg_pos={0 0}, upload=543695]: Done uploading piece. err: <nil> Dec 31 11:56:14.139917508 (620h27m14.011000543s) [noise=true , address=145.79.227.46:4619 , piece_id=BV3NLVQZM457CG2ZYJYZANIDKDVYCQWC64FZGRXFFDV6FFLC2SJA, seg_pos={0 0}, upload=543695]: Done uploading piece. err: <nil> Dec 31 11:56:14.472160101 (620h27m14.343243136s) [noise=true , address=77.90.31.17:4794 , piece_id=ZCNDE7ID7HX43BDZ7KFQKZ6R5V5KUPZWCIRRUQNK65I73JMZQ2JQ, seg_pos={0 0}, upload=543695]: Done uploading piece. err: <nil> Dec 31 11:56:14.476987874 (620h27m14.348070919s) [noise=true , address=45.91.116.5:4326 , piece_id=VDLNHILWQD6ZTD7JMHQY5BDH2A4IHSJTVHJGDI4IK3OBDZSLOADA, seg_pos={0 0}, upload=543695]: Done uploading piece. err: <nil> Dec 31 11:56:14.518004714 (620h27m14.389087749s) [noise=true , address=45.91.116.7:4013 , piece_id=A4VDKYMR74JC325EFEP22V3TOGZRMTQB4UOE5I2Y7JKQF32AIZAA, seg_pos={0 0}, upload=543695]: Done uploading piece. err: <nil> Dec 31 11:56:14.540552146 (620h27m14.411635191s) [noise=true , address=77.90.31.18:4812 , piece_id=KOGGG55EKOFTTT7OOZ5AULZHU2POHF5DMMSX2RM5M65FC3Y3AXRQ, seg_pos={0 0}, upload=543695]: Done uploading piece. err: <nil> Dec 31 11:56:14.677369210 (620h27m14.548452255s) [noise=true , address=145.79.227.40:4058 , piece_id=F3RLQOLC5C6JRKLBZICBMGRSJENDRTTP3PGJEFAOBXNHZP4QI3IA, seg_pos={0 0}, upload=543695]: Done uploading piece. err: <nil> Dec 31 11:56:15.034375707 (620h27m14.905458742s) [noise=true , address=145.79.227.44:4206 , piece_id=TR747XVZACWUD6WGRSROIO5OHFLVBM3XT3XTEWA74GXNXMMYMQHQ, seg_pos={0 0}, upload=543695]: Done uploading piece. err: <nil> Dec 31 11:56:15.095619932 (620h27m14.966702967s) [noise=true , address=145.79.227.49:4358 , piece_id=RP2Y6L6CZD3XOL7XOU3HRDSYZZ6AENXT7LM6XVHZPIZAHSL26BPQ, seg_pos={0 0}, upload=543695]: Done uploading piece. err: <nil> Dec 31 11:56:15.099851950 (620h27m14.970934995s) [noise=true , address=145.79.227.43:4164 , piece_id=QQ4D2S4XAMIDBDSZNYM2G5WT4V7DR4E5WHDSYVT4SZ4YHANAY4QA, seg_pos={0 0}, upload=543695]: Done uploading piece. err: <nil> Dec 31 11:56:15.113070049 (620h27m14.984153094s) [noise=true , address=145.79.227.33:4894 , piece_id=TFZVFOU3GA45OWXOWVA4R2A3W47CS6DCK5K4DAMYYMIH4J5CD23A, seg_pos={0 0}, upload=543695]: Done uploading piece. err: <nil> Dec 31 11:56:15.127931066 (620h27m14.999014101s) [noise=true , address=145.79.227.30:4792 , piece_id=B72NRZMLRWZQRV222OBU6EPXZYYF4K6PXT6L77XZKNVZOA25NLIQ, seg_pos={0 0}, upload=543695]: Done uploading piece. err: <nil> Dec 31 11:56:15.147826669 (620h27m15.018909714s) [noise=true , address=193.149.164.143:4695 , piece_id=5CJUMEUXTJVQOCSHQKZD4ZHQBTFSMUPIQCI7QMJX2URBSUAIGIFA, seg_pos={0 0}, upload=543695]: Done uploading piece. err: <nil> Dec 31 11:56:15.147845695 (620h27m15.01892874s) [seg_pos={0 0}, upload=543695]: Segment reached optimal threshold of 14 pieces. Dec 31 11:56:15.148175553 (620h27m15.019258598s) [noise=true , address=145.79.227.34:4939 , piece_id=TSCIX7EBMZ72GAOPGWHMCUW4PVEONBZ56A3ODVMIMY6LZZW2XDAQ, seg_pos={0 0}, upload=543695]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1h7xrPMs5kHtjm7hiB4hfDaZUHFgtJNoMGqdy3HFif2Aned8eu): protocol: expected piece hash; context canceled; EOF Dec 31 11:56:15.148194439 (620h27m15.019277484s) [noise=true , address=145.79.227.34:4939 , piece_id=TSCIX7EBMZ72GAOPGWHMCUW4PVEONBZ56A3ODVMIMY6LZZW2XDAQ, seg_pos={0 0}, upload=543695]: Long tail cancellation due to optimal threshold Dec 31 11:56:15.148491545 (620h27m15.01957458s) [noise=true , address=77.90.31.13:4657 , piece_id=KSNYKBDG4YOLLPTKC4OFSQR6XOBOGNA64PC3BCKQLRLYGCWFCUUQ, seg_pos={0 0}, upload=543695]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1LLSnWuMa6W7y2jhFUqXoSPEsTyrFsKi8FQDNXF8Tv7XHbP6yt): protocol: write tcp 188.166.225.170:49614->77.90.31.13:4657: use of closed network connection; write tcp 188.166.225.170:49614->77.90.31.13:4657: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49614->77.90.31.13:4657: use of closed network connection Dec 31 11:56:15.148498648 (620h27m15.019581693s) [noise=true , address=77.90.31.13:4657 , piece_id=KSNYKBDG4YOLLPTKC4OFSQR6XOBOGNA64PC3BCKQLRLYGCWFCUUQ, seg_pos={0 0}, upload=543695]: Long tail cancellation due to optimal threshold Dec 31 11:56:15.148641420 (620h27m15.019724465s) [noise=true , address=45.91.116.11:4145 , piece_id=ZX5FCIOENG6476SS6RDITOCPPCDBXAYOHAOAP4Q545FEBKSSQDDA, seg_pos={0 0}, upload=543695]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WKV8PJxq5Kyb3ubCbrfbyYaX6fvRkPMKW2ssUteMFNHm9fP4E): protocol: expected piece hash; context canceled; EOF Dec 31 11:56:15.148647441 (620h27m15.019730476s) [noise=true , address=45.91.116.11:4145 , piece_id=ZX5FCIOENG6476SS6RDITOCPPCDBXAYOHAOAP4Q545FEBKSSQDDA, seg_pos={0 0}, upload=543695]: Long tail cancellation due to optimal threshold Dec 31 11:56:15.148750738 (620h27m15.019833773s) [noise=true , address=64.49.13.169:4707 , piece_id=ZQTZNZZQFGHXU2DIVBDUMJTYRAWAPRATLOGQ6AY4A6N45276NDXQ, seg_pos={0 0}, upload=543695]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:45002->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:45002->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45002->64.49.13.169:4707: use of closed network connection Dec 31 11:56:15.148766589 (620h27m15.019849634s) [noise=true , address=64.49.13.169:4707 , piece_id=ZQTZNZZQFGHXU2DIVBDUMJTYRAWAPRATLOGQ6AY4A6N45276NDXQ, seg_pos={0 0}, upload=543695]: Long tail cancellation due to optimal threshold Dec 31 11:56:15.148782148 (620h27m15.019865193s) [seg_pos={0 0}, upload=543695]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 11:56:15.148845759 (620h27m15.019928804s) [upload=543695]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 11:56:15.148850428 (620h27m15.019933463s) [upload=543695]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 11:56:15.442319734 (620h27m15.313402779s) [upload=543695]: Done uploading segments... Dec 31 11:56:15.442337046 (620h27m15.313420081s) [upload=543695]: Upload finished. err: <nil> Duration: 38.023548549s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [....X..X..XX....X....X.....XX.XX.X....XX.....X.....X....XX..] duration: 6.10160964s trace: https://jaeger.dev.storj.io/trace/7019fd93cdb00dc6 FAIL (2025-12-31 12:09:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, Piece ID: EDYF4XRKLTRTIW4LUHX2MKBXDHB6CYT2P7FO6ZJPM623BOFWMVYA) context deadline exceeded; piecestore: (Node ID: 1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, Piece ID: KHMSB3E6X2FYMNPZ4DLBLXI4FH3O22TICPFDWGTJAGAKKRAVHZAQ) context deadline exceeded ======LOGS====== 1 12:07:11.861573379 (620h38m11.732656424s) [upload=543853]: Starting upload Dec 31 12:07:11.861635807 (620h38m11.732718842s) [upload=543853]: Done starting upload Dec 31 12:07:11.861750156 (620h38m11.732833201s) [upload=543853]: Uploading segments... Dec 31 12:07:11.861836680 (620h38m11.732919725s) [upload=543853]: Got next segment. Inline: false Dec 31 12:07:11.861868741 (620h38m11.732951786s) [upload=543853]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 12:07:11.861874061 (620h38m11.732957106s) [upload=543853]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 12:07:12.105514432 (620h38m11.976597467s) [seg_pos={0 0}, upload=543853]: Begin upload segment... Dec 31 12:07:12.105556783 (620h38m11.976639818s) [seg_pos={0 0}, upload=543853]: Done begin upload segment. Dec 31 12:07:12.105585979 (620h38m11.976669024s) [noise=true , address=145.79.227.2:4115 , piece_id=XKLPIJXVH654UTNUMQDADXOLCNVZQT46UTGCNDYNLPMIUADUCPBA, seg_pos={0 0}, upload=543853]: Uploading piece... Dec 31 12:07:12.105610175 (620h38m11.97669325s) [noise=true , address=145.79.227.17:4971 , piece_id=PH2JN2DOEDDW7KIRMPC35ONPQFFTXBFP5YQBA3N4ISH2THYZDMIA, seg_pos={0 0}, upload=543853]: Uploading piece... Dec 31 12:07:12.106090158 (620h38m11.977173193s) [noise=true , address=145.79.227.22:4509 , piece_id=XNNKDMKGP7AD4JSVOEPBNB6WC6X23B2YLGEK6IBGMLS4BCIA43EA, seg_pos={0 0}, upload=543853]: Uploading piece... Dec 31 12:07:12.106456676 (620h38m11.977539722s) [noise=true , address=77.90.31.11:4554 , piece_id=KHMSB3E6X2FYMNPZ4DLBLXI4FH3O22TICPFDWGTJAGAKKRAVHZAQ, seg_pos={0 0}, upload=543853]: Uploading piece... Dec 31 12:07:12.106742721 (620h38m11.977825766s) [noise=true , address=77.90.31.21:4878 , piece_id=EDYF4XRKLTRTIW4LUHX2MKBXDHB6CYT2P7FO6ZJPM623BOFWMVYA, seg_pos={0 0}, upload=543853]: Uploading piece... Dec 31 12:07:12.107171507 (620h38m11.978254542s) [noise=true , address=145.79.227.50:4374 , piece_id=IPDLL47PNRLUPJIGNPVRH2KOEYALGZVGJLP2M3C4KE35OAK4DVDQ, seg_pos={0 0}, upload=543853]: Uploading piece... Dec 31 12:07:12.107477240 (620h38m11.978560275s) [noise=true , address=145.79.227.4:4187 , piece_id=2DP76MR3IBTSG56BQ2RIGPKUSOABTRXBCNKROP4IHWMSCHKTSCAQ, seg_pos={0 0}, upload=543853]: Uploading piece... Dec 31 12:07:12.107847826 (620h38m11.978930861s) [noise=true , address=145.79.227.15:4874 , piece_id=RXNAVDXFC3TJQZBAJHEJ2MYHQUOXV3NV36KREZ2S7JFYYGFZ6PRA, seg_pos={0 0}, upload=543853]: Uploading piece... Dec 31 12:07:12.108205035 (620h38m11.979288071s) [noise=true , address=145.79.227.37:4566 , piece_id=MITYMLPQAJF2JJSQ4DKDVQDD3RPP7XK6AI4LHCU54XDM5NSZMOKQ, seg_pos={0 0}, upload=543853]: Uploading piece... Dec 31 12:07:12.109451609 (620h38m11.980534654s) [noise=true , address=145.79.227.16:4909 , piece_id=SAELWRFMEC3GDG5TY7JSAVEUPFPAFADTNURLWMGXD6ZYLSTP6YEA, seg_pos={0 0}, upload=543853]: Uploading piece... Dec 31 12:07:12.111851349 (620h38m11.982934394s) [noise=true , address=145.79.227.13:4802 , piece_id=DALBEF23TMYEQWXQ55JM5J2QDO4L6JIEAFBIFWHMK5WTE7MYDFUA, seg_pos={0 0}, upload=543853]: Uploading piece... Dec 31 12:07:12.113438611 (620h38m11.984521646s) [noise=true , address=77.90.31.19:4844 , piece_id=5KPGFPID7GFTLD226A7EML6LSMMEADIJLHSBREG4T2BVTL3IYHJQ, seg_pos={0 0}, upload=543853]: Uploading piece... Dec 31 12:07:12.114694452 (620h38m11.985777487s) [noise=true , address=145.79.227.43:4159 , piece_id=FRTB2LLAAIJFE3DJMK4L4XLCAWZ2BPP23BASHBGMIAWAVMNK5TQQ, seg_pos={0 0}, upload=543853]: Uploading piece... Dec 31 12:07:12.116052859 (620h38m11.987135904s) [noise=true , address=45.91.116.10:4081 , piece_id=NWOEAIHIURDVOHQUO3XV4MEFNTXJEA4L6P7IU6EVOXOQBLNPACHQ, seg_pos={0 0}, upload=543853]: Uploading piece... Dec 31 12:07:12.116061655 (620h38m11.98714469s) [noise=true , address=145.79.227.25:4619 , piece_id=MXSXTGWAEH6PENQUUHXRZ4APEK37U7BLSIXKXJETEDEQGEJFSEKA, seg_pos={0 0}, upload=543853]: Uploading piece... Dec 31 12:07:12.117313990 (620h38m11.988397035s) [noise=true , address=45.91.116.11:4118 , piece_id=QSSUS2PZCFGXIUWMXL4ZXRVORXHN5UYA7POTFQD66DFKHGKR7CJA, seg_pos={0 0}, upload=543853]: Uploading piece... Dec 31 12:07:13.828739360 (620h38m13.699822405s) [upload=543853]: Next returned nil segment Dec 31 12:07:13.828760559 (620h38m13.699843594s) [upload=543853]: Waiting for error group managing segments... Dec 31 12:07:14.107262421 (620h38m13.978345456s) [noise=true , address=145.79.227.4:4187 , piece_id=2DP76MR3IBTSG56BQ2RIGPKUSOABTRXBCNKROP4IHWMSCHKTSCAQ, seg_pos={0 0}, upload=543853]: Done uploading piece. err: <nil> Dec 31 12:07:14.117350740 (620h38m13.988433775s) [noise=true , address=145.79.227.50:4374 , piece_id=IPDLL47PNRLUPJIGNPVRH2KOEYALGZVGJLP2M3C4KE35OAK4DVDQ, seg_pos={0 0}, upload=543853]: Done uploading piece. err: <nil> Dec 31 12:07:14.118644733 (620h38m13.989727769s) [noise=true , address=145.79.227.2:4115 , piece_id=XKLPIJXVH654UTNUMQDADXOLCNVZQT46UTGCNDYNLPMIUADUCPBA, seg_pos={0 0}, upload=543853]: Done uploading piece. err: <nil> Dec 31 12:07:14.119417244 (620h38m13.990500279s) [noise=true , address=145.79.227.25:4619 , piece_id=MXSXTGWAEH6PENQUUHXRZ4APEK37U7BLSIXKXJETEDEQGEJFSEKA, seg_pos={0 0}, upload=543853]: Done uploading piece. err: <nil> Dec 31 12:07:14.122045620 (620h38m13.993128655s) [noise=true , address=145.79.227.17:4971 , piece_id=PH2JN2DOEDDW7KIRMPC35ONPQFFTXBFP5YQBA3N4ISH2THYZDMIA, seg_pos={0 0}, upload=543853]: Done uploading piece. err: <nil> Dec 31 12:07:14.128858542 (620h38m13.999941577s) [noise=true , address=145.79.227.16:4909 , piece_id=SAELWRFMEC3GDG5TY7JSAVEUPFPAFADTNURLWMGXD6ZYLSTP6YEA, seg_pos={0 0}, upload=543853]: Done uploading piece. err: <nil> Dec 31 12:07:14.132529763 (620h38m14.003612798s) [noise=true , address=145.79.227.37:4566 , piece_id=MITYMLPQAJF2JJSQ4DKDVQDD3RPP7XK6AI4LHCU54XDM5NSZMOKQ, seg_pos={0 0}, upload=543853]: Done uploading piece. err: <nil> Dec 31 12:07:14.135029943 (620h38m14.006112988s) [noise=true , address=145.79.227.43:4159 , piece_id=FRTB2LLAAIJFE3DJMK4L4XLCAWZ2BPP23BASHBGMIAWAVMNK5TQQ, seg_pos={0 0}, upload=543853]: Done uploading piece. err: <nil> Dec 31 12:07:14.182526382 (620h38m14.053609427s) [noise=true , address=145.79.227.22:4509 , piece_id=XNNKDMKGP7AD4JSVOEPBNB6WC6X23B2YLGEK6IBGMLS4BCIA43EA, seg_pos={0 0}, upload=543853]: Done uploading piece. err: <nil> Dec 31 12:07:14.191420678 (620h38m14.062503723s) [noise=true , address=145.79.227.13:4802 , piece_id=DALBEF23TMYEQWXQ55JM5J2QDO4L6JIEAFBIFWHMK5WTE7MYDFUA, seg_pos={0 0}, upload=543853]: Done uploading piece. err: <nil> Dec 31 12:07:14.192442173 (620h38m14.063525218s) [noise=true , address=145.79.227.15:4874 , piece_id=RXNAVDXFC3TJQZBAJHEJ2MYHQUOXV3NV36KREZ2S7JFYYGFZ6PRA, seg_pos={0 0}, upload=543853]: Done uploading piece. err: <nil> Dec 31 12:07:14.290745965 (620h38m14.16182903s) [noise=true , address=77.90.31.11:4554 , piece_id=KHMSB3E6X2FYMNPZ4DLBLXI4FH3O22TICPFDWGTJAGAKKRAVHZAQ, seg_pos={0 0}, upload=543853]: Done uploading piece. err: <nil> Dec 31 12:07:14.395876316 (620h38m14.266959361s) [noise=true , address=77.90.31.21:4878 , piece_id=EDYF4XRKLTRTIW4LUHX2MKBXDHB6CYT2P7FO6ZJPM623BOFWMVYA, seg_pos={0 0}, upload=543853]: Done uploading piece. err: <nil> Dec 31 12:07:14.395903037 (620h38m14.266986083s) [seg_pos={0 0}, upload=543853]: Segment reached optimal threshold of 13 pieces. Dec 31 12:07:14.396128787 (620h38m14.267211822s) [noise=true , address=45.91.116.11:4118 , piece_id=QSSUS2PZCFGXIUWMXL4ZXRVORXHN5UYA7POTFQD66DFKHGKR7CJA, seg_pos={0 0}, upload=543853]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: expected piece hash; context canceled; EOF Dec 31 12:07:14.396139528 (620h38m14.267222563s) [noise=true , address=45.91.116.11:4118 , piece_id=QSSUS2PZCFGXIUWMXL4ZXRVORXHN5UYA7POTFQD66DFKHGKR7CJA, seg_pos={0 0}, upload=543853]: Long tail cancellation due to optimal threshold Dec 31 12:07:14.396298831 (620h38m14.267381866s) [noise=true , address=45.91.116.10:4081 , piece_id=NWOEAIHIURDVOHQUO3XV4MEFNTXJEA4L6P7IU6EVOXOQBLNPACHQ, seg_pos={0 0}, upload=543853]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: expected piece hash; context canceled; EOF Dec 31 12:07:14.396308540 (620h38m14.267391585s) [noise=true , address=45.91.116.10:4081 , piece_id=NWOEAIHIURDVOHQUO3XV4MEFNTXJEA4L6P7IU6EVOXOQBLNPACHQ, seg_pos={0 0}, upload=543853]: Long tail cancellation due to optimal threshold Dec 31 12:07:14.396476529 (620h38m14.267559564s) [noise=true , address=77.90.31.19:4844 , piece_id=5KPGFPID7GFTLD226A7EML6LSMMEADIJLHSBREG4T2BVTL3IYHJQ, seg_pos={0 0}, upload=543853]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: write tcp 188.166.225.170:37356->77.90.31.19:4844: use of closed network connection; write tcp 188.166.225.170:37356->77.90.31.19:4844: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37356->77.90.31.19:4844: use of closed network connection Dec 31 12:07:14.396490626 (620h38m14.267573692s) [noise=true , address=77.90.31.19:4844 , piece_id=5KPGFPID7GFTLD226A7EML6LSMMEADIJLHSBREG4T2BVTL3IYHJQ, seg_pos={0 0}, upload=543853]: Long tail cancellation due to optimal threshold Dec 31 12:07:14.396519712 (620h38m14.267602757s) [seg_pos={0 0}, upload=543853]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 12:07:14.396606867 (620h38m14.267689913s) [upload=543853]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 12:07:14.396613249 (620h38m14.267696294s) [upload=543853]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 12:07:14.691428722 (620h38m14.562511757s) [upload=543853]: Done uploading segments... Dec 31 12:07:14.691447397 (620h38m14.562530432s) [upload=543853]: Upload finished. err: <nil> Duration: 6.10164038s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................................X...X......................] duration: 17.38074915s trace: https://jaeger.dev.storj.io/trace/2e598bd65e5b1d02 FAIL (2025-12-31 12:08:29 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 17.380772324s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [.....X..X..XX....X....X.....XX.XX.X....XX.....X.....X....XX.] duration: 5.998142779s trace: https://jaeger.dev.storj.io/trace/7019fd93cdb00dc6 FAIL (2025-12-31 12:08:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, Piece ID: EDYF4XRKLTRTIW4LUHX2MKBXDHB6CYT2P7FO6ZJPM623BOFWMVYA) context deadline exceeded; piecestore: (Node ID: 1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, Piece ID: KHMSB3E6X2FYMNPZ4DLBLXI4FH3O22TICPFDWGTJAGAKKRAVHZAQ) context deadline exceeded ======LOGS====== 1 12:07:11.861573379 (620h38m11.732656424s) [upload=543853]: Starting upload Dec 31 12:07:11.861635807 (620h38m11.732718842s) [upload=543853]: Done starting upload Dec 31 12:07:11.861750156 (620h38m11.732833201s) [upload=543853]: Uploading segments... Dec 31 12:07:11.861836680 (620h38m11.732919725s) [upload=543853]: Got next segment. Inline: false Dec 31 12:07:11.861868741 (620h38m11.732951786s) [upload=543853]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 12:07:11.861874061 (620h38m11.732957106s) [upload=543853]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 12:07:12.105514432 (620h38m11.976597467s) [seg_pos={0 0}, upload=543853]: Begin upload segment... Dec 31 12:07:12.105556783 (620h38m11.976639818s) [seg_pos={0 0}, upload=543853]: Done begin upload segment. Dec 31 12:07:12.105585979 (620h38m11.976669024s) [noise=true , address=145.79.227.2:4115 , piece_id=XKLPIJXVH654UTNUMQDADXOLCNVZQT46UTGCNDYNLPMIUADUCPBA, seg_pos={0 0}, upload=543853]: Uploading piece... Dec 31 12:07:12.105610175 (620h38m11.97669325s) [noise=true , address=145.79.227.17:4971 , piece_id=PH2JN2DOEDDW7KIRMPC35ONPQFFTXBFP5YQBA3N4ISH2THYZDMIA, seg_pos={0 0}, upload=543853]: Uploading piece... Dec 31 12:07:12.106090158 (620h38m11.977173193s) [noise=true , address=145.79.227.22:4509 , piece_id=XNNKDMKGP7AD4JSVOEPBNB6WC6X23B2YLGEK6IBGMLS4BCIA43EA, seg_pos={0 0}, upload=543853]: Uploading piece... Dec 31 12:07:12.106456676 (620h38m11.977539722s) [noise=true , address=77.90.31.11:4554 , piece_id=KHMSB3E6X2FYMNPZ4DLBLXI4FH3O22TICPFDWGTJAGAKKRAVHZAQ, seg_pos={0 0}, upload=543853]: Uploading piece... Dec 31 12:07:12.106742721 (620h38m11.977825766s) [noise=true , address=77.90.31.21:4878 , piece_id=EDYF4XRKLTRTIW4LUHX2MKBXDHB6CYT2P7FO6ZJPM623BOFWMVYA, seg_pos={0 0}, upload=543853]: Uploading piece... Dec 31 12:07:12.107171507 (620h38m11.978254542s) [noise=true , address=145.79.227.50:4374 , piece_id=IPDLL47PNRLUPJIGNPVRH2KOEYALGZVGJLP2M3C4KE35OAK4DVDQ, seg_pos={0 0}, upload=543853]: Uploading piece... Dec 31 12:07:12.107477240 (620h38m11.978560275s) [noise=true , address=145.79.227.4:4187 , piece_id=2DP76MR3IBTSG56BQ2RIGPKUSOABTRXBCNKROP4IHWMSCHKTSCAQ, seg_pos={0 0}, upload=543853]: Uploading piece... Dec 31 12:07:12.107847826 (620h38m11.978930861s) [noise=true , address=145.79.227.15:4874 , piece_id=RXNAVDXFC3TJQZBAJHEJ2MYHQUOXV3NV36KREZ2S7JFYYGFZ6PRA, seg_pos={0 0}, upload=543853]: Uploading piece... Dec 31 12:07:12.108205035 (620h38m11.979288071s) [noise=true , address=145.79.227.37:4566 , piece_id=MITYMLPQAJF2JJSQ4DKDVQDD3RPP7XK6AI4LHCU54XDM5NSZMOKQ, seg_pos={0 0}, upload=543853]: Uploading piece... Dec 31 12:07:12.109451609 (620h38m11.980534654s) [noise=true , address=145.79.227.16:4909 , piece_id=SAELWRFMEC3GDG5TY7JSAVEUPFPAFADTNURLWMGXD6ZYLSTP6YEA, seg_pos={0 0}, upload=543853]: Uploading piece... Dec 31 12:07:12.111851349 (620h38m11.982934394s) [noise=true , address=145.79.227.13:4802 , piece_id=DALBEF23TMYEQWXQ55JM5J2QDO4L6JIEAFBIFWHMK5WTE7MYDFUA, seg_pos={0 0}, upload=543853]: Uploading piece... Dec 31 12:07:12.113438611 (620h38m11.984521646s) [noise=true , address=77.90.31.19:4844 , piece_id=5KPGFPID7GFTLD226A7EML6LSMMEADIJLHSBREG4T2BVTL3IYHJQ, seg_pos={0 0}, upload=543853]: Uploading piece... Dec 31 12:07:12.114694452 (620h38m11.985777487s) [noise=true , address=145.79.227.43:4159 , piece_id=FRTB2LLAAIJFE3DJMK4L4XLCAWZ2BPP23BASHBGMIAWAVMNK5TQQ, seg_pos={0 0}, upload=543853]: Uploading piece... Dec 31 12:07:12.116052859 (620h38m11.987135904s) [noise=true , address=45.91.116.10:4081 , piece_id=NWOEAIHIURDVOHQUO3XV4MEFNTXJEA4L6P7IU6EVOXOQBLNPACHQ, seg_pos={0 0}, upload=543853]: Uploading piece... Dec 31 12:07:12.116061655 (620h38m11.98714469s) [noise=true , address=145.79.227.25:4619 , piece_id=MXSXTGWAEH6PENQUUHXRZ4APEK37U7BLSIXKXJETEDEQGEJFSEKA, seg_pos={0 0}, upload=543853]: Uploading piece... Dec 31 12:07:12.117313990 (620h38m11.988397035s) [noise=true , address=45.91.116.11:4118 , piece_id=QSSUS2PZCFGXIUWMXL4ZXRVORXHN5UYA7POTFQD66DFKHGKR7CJA, seg_pos={0 0}, upload=543853]: Uploading piece... Dec 31 12:07:13.828739360 (620h38m13.699822405s) [upload=543853]: Next returned nil segment Dec 31 12:07:13.828760559 (620h38m13.699843594s) [upload=543853]: Waiting for error group managing segments... Dec 31 12:07:14.107262421 (620h38m13.978345456s) [noise=true , address=145.79.227.4:4187 , piece_id=2DP76MR3IBTSG56BQ2RIGPKUSOABTRXBCNKROP4IHWMSCHKTSCAQ, seg_pos={0 0}, upload=543853]: Done uploading piece. err: <nil> Dec 31 12:07:14.117350740 (620h38m13.988433775s) [noise=true , address=145.79.227.50:4374 , piece_id=IPDLL47PNRLUPJIGNPVRH2KOEYALGZVGJLP2M3C4KE35OAK4DVDQ, seg_pos={0 0}, upload=543853]: Done uploading piece. err: <nil> Dec 31 12:07:14.118644733 (620h38m13.989727769s) [noise=true , address=145.79.227.2:4115 , piece_id=XKLPIJXVH654UTNUMQDADXOLCNVZQT46UTGCNDYNLPMIUADUCPBA, seg_pos={0 0}, upload=543853]: Done uploading piece. err: <nil> Dec 31 12:07:14.119417244 (620h38m13.990500279s) [noise=true , address=145.79.227.25:4619 , piece_id=MXSXTGWAEH6PENQUUHXRZ4APEK37U7BLSIXKXJETEDEQGEJFSEKA, seg_pos={0 0}, upload=543853]: Done uploading piece. err: <nil> Dec 31 12:07:14.122045620 (620h38m13.993128655s) [noise=true , address=145.79.227.17:4971 , piece_id=PH2JN2DOEDDW7KIRMPC35ONPQFFTXBFP5YQBA3N4ISH2THYZDMIA, seg_pos={0 0}, upload=543853]: Done uploading piece. err: <nil> Dec 31 12:07:14.128858542 (620h38m13.999941577s) [noise=true , address=145.79.227.16:4909 , piece_id=SAELWRFMEC3GDG5TY7JSAVEUPFPAFADTNURLWMGXD6ZYLSTP6YEA, seg_pos={0 0}, upload=543853]: Done uploading piece. err: <nil> Dec 31 12:07:14.132529763 (620h38m14.003612798s) [noise=true , address=145.79.227.37:4566 , piece_id=MITYMLPQAJF2JJSQ4DKDVQDD3RPP7XK6AI4LHCU54XDM5NSZMOKQ, seg_pos={0 0}, upload=543853]: Done uploading piece. err: <nil> Dec 31 12:07:14.135029943 (620h38m14.006112988s) [noise=true , address=145.79.227.43:4159 , piece_id=FRTB2LLAAIJFE3DJMK4L4XLCAWZ2BPP23BASHBGMIAWAVMNK5TQQ, seg_pos={0 0}, upload=543853]: Done uploading piece. err: <nil> Dec 31 12:07:14.182526382 (620h38m14.053609427s) [noise=true , address=145.79.227.22:4509 , piece_id=XNNKDMKGP7AD4JSVOEPBNB6WC6X23B2YLGEK6IBGMLS4BCIA43EA, seg_pos={0 0}, upload=543853]: Done uploading piece. err: <nil> Dec 31 12:07:14.191420678 (620h38m14.062503723s) [noise=true , address=145.79.227.13:4802 , piece_id=DALBEF23TMYEQWXQ55JM5J2QDO4L6JIEAFBIFWHMK5WTE7MYDFUA, seg_pos={0 0}, upload=543853]: Done uploading piece. err: <nil> Dec 31 12:07:14.192442173 (620h38m14.063525218s) [noise=true , address=145.79.227.15:4874 , piece_id=RXNAVDXFC3TJQZBAJHEJ2MYHQUOXV3NV36KREZ2S7JFYYGFZ6PRA, seg_pos={0 0}, upload=543853]: Done uploading piece. err: <nil> Dec 31 12:07:14.290745965 (620h38m14.16182903s) [noise=true , address=77.90.31.11:4554 , piece_id=KHMSB3E6X2FYMNPZ4DLBLXI4FH3O22TICPFDWGTJAGAKKRAVHZAQ, seg_pos={0 0}, upload=543853]: Done uploading piece. err: <nil> Dec 31 12:07:14.395876316 (620h38m14.266959361s) [noise=true , address=77.90.31.21:4878 , piece_id=EDYF4XRKLTRTIW4LUHX2MKBXDHB6CYT2P7FO6ZJPM623BOFWMVYA, seg_pos={0 0}, upload=543853]: Done uploading piece. err: <nil> Dec 31 12:07:14.395903037 (620h38m14.266986083s) [seg_pos={0 0}, upload=543853]: Segment reached optimal threshold of 13 pieces. Dec 31 12:07:14.396128787 (620h38m14.267211822s) [noise=true , address=45.91.116.11:4118 , piece_id=QSSUS2PZCFGXIUWMXL4ZXRVORXHN5UYA7POTFQD66DFKHGKR7CJA, seg_pos={0 0}, upload=543853]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: expected piece hash; context canceled; EOF Dec 31 12:07:14.396139528 (620h38m14.267222563s) [noise=true , address=45.91.116.11:4118 , piece_id=QSSUS2PZCFGXIUWMXL4ZXRVORXHN5UYA7POTFQD66DFKHGKR7CJA, seg_pos={0 0}, upload=543853]: Long tail cancellation due to optimal threshold Dec 31 12:07:14.396298831 (620h38m14.267381866s) [noise=true , address=45.91.116.10:4081 , piece_id=NWOEAIHIURDVOHQUO3XV4MEFNTXJEA4L6P7IU6EVOXOQBLNPACHQ, seg_pos={0 0}, upload=543853]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: expected piece hash; context canceled; EOF Dec 31 12:07:14.396308540 (620h38m14.267391585s) [noise=true , address=45.91.116.10:4081 , piece_id=NWOEAIHIURDVOHQUO3XV4MEFNTXJEA4L6P7IU6EVOXOQBLNPACHQ, seg_pos={0 0}, upload=543853]: Long tail cancellation due to optimal threshold Dec 31 12:07:14.396476529 (620h38m14.267559564s) [noise=true , address=77.90.31.19:4844 , piece_id=5KPGFPID7GFTLD226A7EML6LSMMEADIJLHSBREG4T2BVTL3IYHJQ, seg_pos={0 0}, upload=543853]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: write tcp 188.166.225.170:37356->77.90.31.19:4844: use of closed network connection; write tcp 188.166.225.170:37356->77.90.31.19:4844: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37356->77.90.31.19:4844: use of closed network connection Dec 31 12:07:14.396490626 (620h38m14.267573692s) [noise=true , address=77.90.31.19:4844 , piece_id=5KPGFPID7GFTLD226A7EML6LSMMEADIJLHSBREG4T2BVTL3IYHJQ, seg_pos={0 0}, upload=543853]: Long tail cancellation due to optimal threshold Dec 31 12:07:14.396519712 (620h38m14.267602757s) [seg_pos={0 0}, upload=543853]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 12:07:14.396606867 (620h38m14.267689913s) [upload=543853]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 12:07:14.396613249 (620h38m14.267696294s) [upload=543853]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 12:07:14.691428722 (620h38m14.562511757s) [upload=543853]: Done uploading segments... Dec 31 12:07:14.691447397 (620h38m14.562530432s) [upload=543853]: Upload finished. err: <nil> Duration: 5.998173588s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [.X................X..........X.X......X.............X.......] duration: 3.517085941s trace: https://jaeger.dev.storj.io/trace/2c96062d1b5b5a44 FAIL (2025-12-31 12:08:22 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.517111139s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [X..............................................X............] duration: 26.928639593s trace: https://jaeger.dev.storj.io/trace/3696061a2156809c FAIL (2025-12-31 12:08:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== .17:4794 , piece_id=ZCNDE7ID7HX43BDZ7KFQKZ6R5V5KUPZWCIRRUQNK65I73JMZQ2JQ, seg_pos={0 0}, upload=543695]: Uploading piece... Dec 31 11:56:12.262721887 (620h27m12.133804932s) [noise=true , address=193.149.164.143:4695 , piece_id=5CJUMEUXTJVQOCSHQKZD4ZHQBTFSMUPIQCI7QMJX2URBSUAIGIFA, seg_pos={0 0}, upload=543695]: Uploading piece... Dec 31 11:56:12.263130455 (620h27m12.1342135s) [noise=true , address=45.91.116.5:4326 , piece_id=VDLNHILWQD6ZTD7JMHQY5BDH2A4IHSJTVHJGDI4IK3OBDZSLOADA, seg_pos={0 0}, upload=543695]: Uploading piece... Dec 31 11:56:12.264175605 (620h27m12.13525865s) [noise=true , address=77.90.31.18:4812 , piece_id=KOGGG55EKOFTTT7OOZ5AULZHU2POHF5DMMSX2RM5M65FC3Y3AXRQ, seg_pos={0 0}, upload=543695]: Uploading piece... Dec 31 11:56:12.264193529 (620h27m12.135276574s) [noise=true , address=145.79.227.49:4358 , piece_id=RP2Y6L6CZD3XOL7XOU3HRDSYZZ6AENXT7LM6XVHZPIZAHSL26BPQ, seg_pos={0 0}, upload=543695]: Uploading piece... Dec 31 11:56:12.264609000 (620h27m12.135692035s) [noise=true , address=45.91.116.11:4145 , piece_id=ZX5FCIOENG6476SS6RDITOCPPCDBXAYOHAOAP4Q545FEBKSSQDDA, seg_pos={0 0}, upload=543695]: Uploading piece... Dec 31 11:56:12.264666079 (620h27m12.135749114s) [noise=true , address=64.49.13.140:4505 , piece_id=YRQ2VLRASSQVB65ONBLZXVLXY7COOSYVUUNMKJO5AOAJKFFIEW2A, seg_pos={0 0}, upload=543695]: Uploading piece... Dec 31 11:56:12.265310566 (620h27m12.136393611s) [noise=true , address=145.79.227.44:4206 , piece_id=TR747XVZACWUD6WGRSROIO5OHFLVBM3XT3XTEWA74GXNXMMYMQHQ, seg_pos={0 0}, upload=543695]: Uploading piece... Dec 31 11:56:12.265745504 (620h27m12.136828549s) [noise=true , address=145.79.227.33:4894 , piece_id=TFZVFOU3GA45OWXOWVA4R2A3W47CS6DCK5K4DAMYYMIH4J5CD23A, seg_pos={0 0}, upload=543695]: Uploading piece... Dec 31 11:56:12.266321861 (620h27m12.137404896s) [noise=true , address=45.91.116.7:4013 , piece_id=A4VDKYMR74JC325EFEP22V3TOGZRMTQB4UOE5I2Y7JKQF32AIZAA, seg_pos={0 0}, upload=543695]: Uploading piece... Dec 31 11:56:12.266868933 (620h27m12.137951978s) [noise=true , address=64.49.13.169:4707 , piece_id=ZQTZNZZQFGHXU2DIVBDUMJTYRAWAPRATLOGQ6AY4A6N45276NDXQ, seg_pos={0 0}, upload=543695]: Uploading piece... Dec 31 11:56:12.267258635 (620h27m12.13834167s) [noise=true , address=77.90.31.13:4657 , piece_id=KSNYKBDG4YOLLPTKC4OFSQR6XOBOGNA64PC3BCKQLRLYGCWFCUUQ, seg_pos={0 0}, upload=543695]: Uploading piece... Dec 31 11:56:12.267587522 (620h27m12.138670567s) [noise=true , address=145.79.227.40:4058 , piece_id=F3RLQOLC5C6JRKLBZICBMGRSJENDRTTP3PGJEFAOBXNHZP4QI3IA, seg_pos={0 0}, upload=543695]: Uploading piece... Dec 31 11:56:12.268440155 (620h27m12.13952319s) [noise=true , address=145.79.227.34:4939 , piece_id=TSCIX7EBMZ72GAOPGWHMCUW4PVEONBZ56A3ODVMIMY6LZZW2XDAQ, seg_pos={0 0}, upload=543695]: Uploading piece... Dec 31 11:56:12.269297889 (620h27m12.140380924s) [noise=true , address=145.79.227.2:4134 , piece_id=HWWXLVVLSMPGUKIP5A5T5IXAB3ZCABH7OA5DIJSNDXRQU23EIFNA, seg_pos={0 0}, upload=543695]: Uploading piece... Dec 31 11:56:13.698835153 (620h27m13.569918188s) [upload=543695]: Next returned nil segment Dec 31 11:56:13.698965801 (620h27m13.570048846s) [upload=543695]: Waiting for error group managing segments... Dec 31 11:56:14.047217086 (620h27m13.918300131s) [noise=true , address=64.49.13.140:4505 , piece_id=YRQ2VLRASSQVB65ONBLZXVLXY7COOSYVUUNMKJO5AOAJKFFIEW2A, seg_pos={0 0}, upload=543695]: Done uploading piece. err: <nil> Dec 31 11:56:14.137538488 (620h27m14.008621523s) [noise=true , address=145.79.227.2:4134 , piece_id=HWWXLVVLSMPGUKIP5A5T5IXAB3ZCABH7OA5DIJSNDXRQU23EIFNA, seg_pos={0 0}, upload=543695]: Done uploading piece. err: <nil> Dec 31 11:56:14.139917508 (620h27m14.011000543s) [noise=true , address=145.79.227.46:4619 , piece_id=BV3NLVQZM457CG2ZYJYZANIDKDVYCQWC64FZGRXFFDV6FFLC2SJA, seg_pos={0 0}, upload=543695]: Done uploading piece. err: <nil> Dec 31 11:56:14.472160101 (620h27m14.343243136s) [noise=true , address=77.90.31.17:4794 , piece_id=ZCNDE7ID7HX43BDZ7KFQKZ6R5V5KUPZWCIRRUQNK65I73JMZQ2JQ, seg_pos={0 0}, upload=543695]: Done uploading piece. err: <nil> Dec 31 11:56:14.476987874 (620h27m14.348070919s) [noise=true , address=45.91.116.5:4326 , piece_id=VDLNHILWQD6ZTD7JMHQY5BDH2A4IHSJTVHJGDI4IK3OBDZSLOADA, seg_pos={0 0}, upload=543695]: Done uploading piece. err: <nil> Dec 31 11:56:14.518004714 (620h27m14.389087749s) [noise=true , address=45.91.116.7:4013 , piece_id=A4VDKYMR74JC325EFEP22V3TOGZRMTQB4UOE5I2Y7JKQF32AIZAA, seg_pos={0 0}, upload=543695]: Done uploading piece. err: <nil> Dec 31 11:56:14.540552146 (620h27m14.411635191s) [noise=true , address=77.90.31.18:4812 , piece_id=KOGGG55EKOFTTT7OOZ5AULZHU2POHF5DMMSX2RM5M65FC3Y3AXRQ, seg_pos={0 0}, upload=543695]: Done uploading piece. err: <nil> Dec 31 11:56:14.677369210 (620h27m14.548452255s) [noise=true , address=145.79.227.40:4058 , piece_id=F3RLQOLC5C6JRKLBZICBMGRSJENDRTTP3PGJEFAOBXNHZP4QI3IA, seg_pos={0 0}, upload=543695]: Done uploading piece. err: <nil> Dec 31 11:56:15.034375707 (620h27m14.905458742s) [noise=true , address=145.79.227.44:4206 , piece_id=TR747XVZACWUD6WGRSROIO5OHFLVBM3XT3XTEWA74GXNXMMYMQHQ, seg_pos={0 0}, upload=543695]: Done uploading piece. err: <nil> Dec 31 11:56:15.095619932 (620h27m14.966702967s) [noise=true , address=145.79.227.49:4358 , piece_id=RP2Y6L6CZD3XOL7XOU3HRDSYZZ6AENXT7LM6XVHZPIZAHSL26BPQ, seg_pos={0 0}, upload=543695]: Done uploading piece. err: <nil> Dec 31 11:56:15.099851950 (620h27m14.970934995s) [noise=true , address=145.79.227.43:4164 , piece_id=QQ4D2S4XAMIDBDSZNYM2G5WT4V7DR4E5WHDSYVT4SZ4YHANAY4QA, seg_pos={0 0}, upload=543695]: Done uploading piece. err: <nil> Dec 31 11:56:15.113070049 (620h27m14.984153094s) [noise=true , address=145.79.227.33:4894 , piece_id=TFZVFOU3GA45OWXOWVA4R2A3W47CS6DCK5K4DAMYYMIH4J5CD23A, seg_pos={0 0}, upload=543695]: Done uploading piece. err: <nil> Dec 31 11:56:15.127931066 (620h27m14.999014101s) [noise=true , address=145.79.227.30:4792 , piece_id=B72NRZMLRWZQRV222OBU6EPXZYYF4K6PXT6L77XZKNVZOA25NLIQ, seg_pos={0 0}, upload=543695]: Done uploading piece. err: <nil> Dec 31 11:56:15.147826669 (620h27m15.018909714s) [noise=true , address=193.149.164.143:4695 , piece_id=5CJUMEUXTJVQOCSHQKZD4ZHQBTFSMUPIQCI7QMJX2URBSUAIGIFA, seg_pos={0 0}, upload=543695]: Done uploading piece. err: <nil> Dec 31 11:56:15.147845695 (620h27m15.01892874s) [seg_pos={0 0}, upload=543695]: Segment reached optimal threshold of 14 pieces. Dec 31 11:56:15.148175553 (620h27m15.019258598s) [noise=true , address=145.79.227.34:4939 , piece_id=TSCIX7EBMZ72GAOPGWHMCUW4PVEONBZ56A3ODVMIMY6LZZW2XDAQ, seg_pos={0 0}, upload=543695]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1h7xrPMs5kHtjm7hiB4hfDaZUHFgtJNoMGqdy3HFif2Aned8eu): protocol: expected piece hash; context canceled; EOF Dec 31 11:56:15.148194439 (620h27m15.019277484s) [noise=true , address=145.79.227.34:4939 , piece_id=TSCIX7EBMZ72GAOPGWHMCUW4PVEONBZ56A3ODVMIMY6LZZW2XDAQ, seg_pos={0 0}, upload=543695]: Long tail cancellation due to optimal threshold Dec 31 11:56:15.148491545 (620h27m15.01957458s) [noise=true , address=77.90.31.13:4657 , piece_id=KSNYKBDG4YOLLPTKC4OFSQR6XOBOGNA64PC3BCKQLRLYGCWFCUUQ, seg_pos={0 0}, upload=543695]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1LLSnWuMa6W7y2jhFUqXoSPEsTyrFsKi8FQDNXF8Tv7XHbP6yt): protocol: write tcp 188.166.225.170:49614->77.90.31.13:4657: use of closed network connection; write tcp 188.166.225.170:49614->77.90.31.13:4657: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49614->77.90.31.13:4657: use of closed network connection Dec 31 11:56:15.148498648 (620h27m15.019581693s) [noise=true , address=77.90.31.13:4657 , piece_id=KSNYKBDG4YOLLPTKC4OFSQR6XOBOGNA64PC3BCKQLRLYGCWFCUUQ, seg_pos={0 0}, upload=543695]: Long tail cancellation due to optimal threshold Dec 31 11:56:15.148641420 (620h27m15.019724465s) [noise=true , address=45.91.116.11:4145 , piece_id=ZX5FCIOENG6476SS6RDITOCPPCDBXAYOHAOAP4Q545FEBKSSQDDA, seg_pos={0 0}, upload=543695]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WKV8PJxq5Kyb3ubCbrfbyYaX6fvRkPMKW2ssUteMFNHm9fP4E): protocol: expected piece hash; context canceled; EOF Dec 31 11:56:15.148647441 (620h27m15.019730476s) [noise=true , address=45.91.116.11:4145 , piece_id=ZX5FCIOENG6476SS6RDITOCPPCDBXAYOHAOAP4Q545FEBKSSQDDA, seg_pos={0 0}, upload=543695]: Long tail cancellation due to optimal threshold Dec 31 11:56:15.148750738 (620h27m15.019833773s) [noise=true , address=64.49.13.169:4707 , piece_id=ZQTZNZZQFGHXU2DIVBDUMJTYRAWAPRATLOGQ6AY4A6N45276NDXQ, seg_pos={0 0}, upload=543695]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:45002->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:45002->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45002->64.49.13.169:4707: use of closed network connection Dec 31 11:56:15.148766589 (620h27m15.019849634s) [noise=true , address=64.49.13.169:4707 , piece_id=ZQTZNZZQFGHXU2DIVBDUMJTYRAWAPRATLOGQ6AY4A6N45276NDXQ, seg_pos={0 0}, upload=543695]: Long tail cancellation due to optimal threshold Dec 31 11:56:15.148782148 (620h27m15.019865193s) [seg_pos={0 0}, upload=543695]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 11:56:15.148845759 (620h27m15.019928804s) [upload=543695]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 11:56:15.148850428 (620h27m15.019933463s) [upload=543695]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 11:56:15.442319734 (620h27m15.313402779s) [upload=543695]: Done uploading segments... Dec 31 11:56:15.442337046 (620h27m15.313420081s) [upload=543695]: Upload finished. err: <nil> Duration: 26.928707945s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................................X...X.....................] duration: 6.523320864s trace: https://jaeger.dev.storj.io/trace/2e598bd65e5b1d02 FAIL (2025-12-31 12:07:42 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.523336325s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [......X..X..XX....X....X.....XX.XX.X....XX.....X.....X....XX] duration: 30.002663942s trace: https://jaeger.dev.storj.io/trace/7019fd93cdb00dc6 FAIL (2025-12-31 12:07:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, Piece ID: EDYF4XRKLTRTIW4LUHX2MKBXDHB6CYT2P7FO6ZJPM623BOFWMVYA) context deadline exceeded; piecestore: (Node ID: 1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, Piece ID: KHMSB3E6X2FYMNPZ4DLBLXI4FH3O22TICPFDWGTJAGAKKRAVHZAQ) context deadline exceeded ======LOGS====== 1 12:07:11.861573379 (620h38m11.732656424s) [upload=543853]: Starting upload Dec 31 12:07:11.861635807 (620h38m11.732718842s) [upload=543853]: Done starting upload Dec 31 12:07:11.861750156 (620h38m11.732833201s) [upload=543853]: Uploading segments... Dec 31 12:07:11.861836680 (620h38m11.732919725s) [upload=543853]: Got next segment. Inline: false Dec 31 12:07:11.861868741 (620h38m11.732951786s) [upload=543853]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 12:07:11.861874061 (620h38m11.732957106s) [upload=543853]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 12:07:12.105514432 (620h38m11.976597467s) [seg_pos={0 0}, upload=543853]: Begin upload segment... Dec 31 12:07:12.105556783 (620h38m11.976639818s) [seg_pos={0 0}, upload=543853]: Done begin upload segment. Dec 31 12:07:12.105585979 (620h38m11.976669024s) [noise=true , address=145.79.227.2:4115 , piece_id=XKLPIJXVH654UTNUMQDADXOLCNVZQT46UTGCNDYNLPMIUADUCPBA, seg_pos={0 0}, upload=543853]: Uploading piece... Dec 31 12:07:12.105610175 (620h38m11.97669325s) [noise=true , address=145.79.227.17:4971 , piece_id=PH2JN2DOEDDW7KIRMPC35ONPQFFTXBFP5YQBA3N4ISH2THYZDMIA, seg_pos={0 0}, upload=543853]: Uploading piece... Dec 31 12:07:12.106090158 (620h38m11.977173193s) [noise=true , address=145.79.227.22:4509 , piece_id=XNNKDMKGP7AD4JSVOEPBNB6WC6X23B2YLGEK6IBGMLS4BCIA43EA, seg_pos={0 0}, upload=543853]: Uploading piece... Dec 31 12:07:12.106456676 (620h38m11.977539722s) [noise=true , address=77.90.31.11:4554 , piece_id=KHMSB3E6X2FYMNPZ4DLBLXI4FH3O22TICPFDWGTJAGAKKRAVHZAQ, seg_pos={0 0}, upload=543853]: Uploading piece... Dec 31 12:07:12.106742721 (620h38m11.977825766s) [noise=true , address=77.90.31.21:4878 , piece_id=EDYF4XRKLTRTIW4LUHX2MKBXDHB6CYT2P7FO6ZJPM623BOFWMVYA, seg_pos={0 0}, upload=543853]: Uploading piece... Dec 31 12:07:12.107171507 (620h38m11.978254542s) [noise=true , address=145.79.227.50:4374 , piece_id=IPDLL47PNRLUPJIGNPVRH2KOEYALGZVGJLP2M3C4KE35OAK4DVDQ, seg_pos={0 0}, upload=543853]: Uploading piece... Dec 31 12:07:12.107477240 (620h38m11.978560275s) [noise=true , address=145.79.227.4:4187 , piece_id=2DP76MR3IBTSG56BQ2RIGPKUSOABTRXBCNKROP4IHWMSCHKTSCAQ, seg_pos={0 0}, upload=543853]: Uploading piece... Dec 31 12:07:12.107847826 (620h38m11.978930861s) [noise=true , address=145.79.227.15:4874 , piece_id=RXNAVDXFC3TJQZBAJHEJ2MYHQUOXV3NV36KREZ2S7JFYYGFZ6PRA, seg_pos={0 0}, upload=543853]: Uploading piece... Dec 31 12:07:12.108205035 (620h38m11.979288071s) [noise=true , address=145.79.227.37:4566 , piece_id=MITYMLPQAJF2JJSQ4DKDVQDD3RPP7XK6AI4LHCU54XDM5NSZMOKQ, seg_pos={0 0}, upload=543853]: Uploading piece... Dec 31 12:07:12.109451609 (620h38m11.980534654s) [noise=true , address=145.79.227.16:4909 , piece_id=SAELWRFMEC3GDG5TY7JSAVEUPFPAFADTNURLWMGXD6ZYLSTP6YEA, seg_pos={0 0}, upload=543853]: Uploading piece... Dec 31 12:07:12.111851349 (620h38m11.982934394s) [noise=true , address=145.79.227.13:4802 , piece_id=DALBEF23TMYEQWXQ55JM5J2QDO4L6JIEAFBIFWHMK5WTE7MYDFUA, seg_pos={0 0}, upload=543853]: Uploading piece... Dec 31 12:07:12.113438611 (620h38m11.984521646s) [noise=true , address=77.90.31.19:4844 , piece_id=5KPGFPID7GFTLD226A7EML6LSMMEADIJLHSBREG4T2BVTL3IYHJQ, seg_pos={0 0}, upload=543853]: Uploading piece... Dec 31 12:07:12.114694452 (620h38m11.985777487s) [noise=true , address=145.79.227.43:4159 , piece_id=FRTB2LLAAIJFE3DJMK4L4XLCAWZ2BPP23BASHBGMIAWAVMNK5TQQ, seg_pos={0 0}, upload=543853]: Uploading piece... Dec 31 12:07:12.116052859 (620h38m11.987135904s) [noise=true , address=45.91.116.10:4081 , piece_id=NWOEAIHIURDVOHQUO3XV4MEFNTXJEA4L6P7IU6EVOXOQBLNPACHQ, seg_pos={0 0}, upload=543853]: Uploading piece... Dec 31 12:07:12.116061655 (620h38m11.98714469s) [noise=true , address=145.79.227.25:4619 , piece_id=MXSXTGWAEH6PENQUUHXRZ4APEK37U7BLSIXKXJETEDEQGEJFSEKA, seg_pos={0 0}, upload=543853]: Uploading piece... Dec 31 12:07:12.117313990 (620h38m11.988397035s) [noise=true , address=45.91.116.11:4118 , piece_id=QSSUS2PZCFGXIUWMXL4ZXRVORXHN5UYA7POTFQD66DFKHGKR7CJA, seg_pos={0 0}, upload=543853]: Uploading piece... Dec 31 12:07:13.828739360 (620h38m13.699822405s) [upload=543853]: Next returned nil segment Dec 31 12:07:13.828760559 (620h38m13.699843594s) [upload=543853]: Waiting for error group managing segments... Dec 31 12:07:14.107262421 (620h38m13.978345456s) [noise=true , address=145.79.227.4:4187 , piece_id=2DP76MR3IBTSG56BQ2RIGPKUSOABTRXBCNKROP4IHWMSCHKTSCAQ, seg_pos={0 0}, upload=543853]: Done uploading piece. err: <nil> Dec 31 12:07:14.117350740 (620h38m13.988433775s) [noise=true , address=145.79.227.50:4374 , piece_id=IPDLL47PNRLUPJIGNPVRH2KOEYALGZVGJLP2M3C4KE35OAK4DVDQ, seg_pos={0 0}, upload=543853]: Done uploading piece. err: <nil> Dec 31 12:07:14.118644733 (620h38m13.989727769s) [noise=true , address=145.79.227.2:4115 , piece_id=XKLPIJXVH654UTNUMQDADXOLCNVZQT46UTGCNDYNLPMIUADUCPBA, seg_pos={0 0}, upload=543853]: Done uploading piece. err: <nil> Dec 31 12:07:14.119417244 (620h38m13.990500279s) [noise=true , address=145.79.227.25:4619 , piece_id=MXSXTGWAEH6PENQUUHXRZ4APEK37U7BLSIXKXJETEDEQGEJFSEKA, seg_pos={0 0}, upload=543853]: Done uploading piece. err: <nil> Dec 31 12:07:14.122045620 (620h38m13.993128655s) [noise=true , address=145.79.227.17:4971 , piece_id=PH2JN2DOEDDW7KIRMPC35ONPQFFTXBFP5YQBA3N4ISH2THYZDMIA, seg_pos={0 0}, upload=543853]: Done uploading piece. err: <nil> Dec 31 12:07:14.128858542 (620h38m13.999941577s) [noise=true , address=145.79.227.16:4909 , piece_id=SAELWRFMEC3GDG5TY7JSAVEUPFPAFADTNURLWMGXD6ZYLSTP6YEA, seg_pos={0 0}, upload=543853]: Done uploading piece. err: <nil> Dec 31 12:07:14.132529763 (620h38m14.003612798s) [noise=true , address=145.79.227.37:4566 , piece_id=MITYMLPQAJF2JJSQ4DKDVQDD3RPP7XK6AI4LHCU54XDM5NSZMOKQ, seg_pos={0 0}, upload=543853]: Done uploading piece. err: <nil> Dec 31 12:07:14.135029943 (620h38m14.006112988s) [noise=true , address=145.79.227.43:4159 , piece_id=FRTB2LLAAIJFE3DJMK4L4XLCAWZ2BPP23BASHBGMIAWAVMNK5TQQ, seg_pos={0 0}, upload=543853]: Done uploading piece. err: <nil> Dec 31 12:07:14.182526382 (620h38m14.053609427s) [noise=true , address=145.79.227.22:4509 , piece_id=XNNKDMKGP7AD4JSVOEPBNB6WC6X23B2YLGEK6IBGMLS4BCIA43EA, seg_pos={0 0}, upload=543853]: Done uploading piece. err: <nil> Dec 31 12:07:14.191420678 (620h38m14.062503723s) [noise=true , address=145.79.227.13:4802 , piece_id=DALBEF23TMYEQWXQ55JM5J2QDO4L6JIEAFBIFWHMK5WTE7MYDFUA, seg_pos={0 0}, upload=543853]: Done uploading piece. err: <nil> Dec 31 12:07:14.192442173 (620h38m14.063525218s) [noise=true , address=145.79.227.15:4874 , piece_id=RXNAVDXFC3TJQZBAJHEJ2MYHQUOXV3NV36KREZ2S7JFYYGFZ6PRA, seg_pos={0 0}, upload=543853]: Done uploading piece. err: <nil> Dec 31 12:07:14.290745965 (620h38m14.16182903s) [noise=true , address=77.90.31.11:4554 , piece_id=KHMSB3E6X2FYMNPZ4DLBLXI4FH3O22TICPFDWGTJAGAKKRAVHZAQ, seg_pos={0 0}, upload=543853]: Done uploading piece. err: <nil> Dec 31 12:07:14.395876316 (620h38m14.266959361s) [noise=true , address=77.90.31.21:4878 , piece_id=EDYF4XRKLTRTIW4LUHX2MKBXDHB6CYT2P7FO6ZJPM623BOFWMVYA, seg_pos={0 0}, upload=543853]: Done uploading piece. err: <nil> Dec 31 12:07:14.395903037 (620h38m14.266986083s) [seg_pos={0 0}, upload=543853]: Segment reached optimal threshold of 13 pieces. Dec 31 12:07:14.396128787 (620h38m14.267211822s) [noise=true , address=45.91.116.11:4118 , piece_id=QSSUS2PZCFGXIUWMXL4ZXRVORXHN5UYA7POTFQD66DFKHGKR7CJA, seg_pos={0 0}, upload=543853]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: expected piece hash; context canceled; EOF Dec 31 12:07:14.396139528 (620h38m14.267222563s) [noise=true , address=45.91.116.11:4118 , piece_id=QSSUS2PZCFGXIUWMXL4ZXRVORXHN5UYA7POTFQD66DFKHGKR7CJA, seg_pos={0 0}, upload=543853]: Long tail cancellation due to optimal threshold Dec 31 12:07:14.396298831 (620h38m14.267381866s) [noise=true , address=45.91.116.10:4081 , piece_id=NWOEAIHIURDVOHQUO3XV4MEFNTXJEA4L6P7IU6EVOXOQBLNPACHQ, seg_pos={0 0}, upload=543853]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: expected piece hash; context canceled; EOF Dec 31 12:07:14.396308540 (620h38m14.267391585s) [noise=true , address=45.91.116.10:4081 , piece_id=NWOEAIHIURDVOHQUO3XV4MEFNTXJEA4L6P7IU6EVOXOQBLNPACHQ, seg_pos={0 0}, upload=543853]: Long tail cancellation due to optimal threshold Dec 31 12:07:14.396476529 (620h38m14.267559564s) [noise=true , address=77.90.31.19:4844 , piece_id=5KPGFPID7GFTLD226A7EML6LSMMEADIJLHSBREG4T2BVTL3IYHJQ, seg_pos={0 0}, upload=543853]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: write tcp 188.166.225.170:37356->77.90.31.19:4844: use of closed network connection; write tcp 188.166.225.170:37356->77.90.31.19:4844: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37356->77.90.31.19:4844: use of closed network connection Dec 31 12:07:14.396490626 (620h38m14.267573692s) [noise=true , address=77.90.31.19:4844 , piece_id=5KPGFPID7GFTLD226A7EML6LSMMEADIJLHSBREG4T2BVTL3IYHJQ, seg_pos={0 0}, upload=543853]: Long tail cancellation due to optimal threshold Dec 31 12:07:14.396519712 (620h38m14.267602757s) [seg_pos={0 0}, upload=543853]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 12:07:14.396606867 (620h38m14.267689913s) [upload=543853]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 12:07:14.396613249 (620h38m14.267696294s) [upload=543853]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 12:07:14.691428722 (620h38m14.562511757s) [upload=543853]: Done uploading segments... Dec 31 12:07:14.691447397 (620h38m14.562530432s) [upload=543853]: Upload finished. err: <nil> Duration: 30.002674992s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................................X...X....................] duration: 25.670934256s trace: https://jaeger.dev.storj.io/trace/2e598bd65e5b1d02 FAIL (2025-12-31 12:07:35 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 25.670961446s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [..X................X..........X.X......X.............X......] duration: 4.321226389s trace: https://jaeger.dev.storj.io/trace/2c96062d1b5b5a44 FAIL (2025-12-31 12:07:25 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.321254282s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [.......X..X..XX....X....X.....XX.XX.X....XX.....X.....X....X] duration: 30.002821432s trace: https://jaeger.dev.storj.io/trace/1bc46e129aa1c658 FAIL (2025-12-31 12:07:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 31 12:06:41.632118750 (620h37m41.503201795s) [upload=543845]: Starting upload Dec 31 12:06:41.632196178 (620h37m41.503279234s) [upload=543845]: Done starting upload Dec 31 12:06:41.632419353 (620h37m41.503502398s) [upload=543845]: Uploading segments... Dec 31 12:06:41.632516659 (620h37m41.503599704s) [upload=543845]: Got next segment. Inline: false Dec 31 12:06:41.632579027 (620h37m41.503662073s) [upload=543845]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 12:06:41.632585480 (620h37m41.503668515s) [upload=543845]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 12:06:41.876547282 (620h37m41.747630327s) [seg_pos={0 0}, upload=543845]: Begin upload segment... Dec 31 12:06:41.876600774 (620h37m41.747683819s) [seg_pos={0 0}, upload=543845]: Done begin upload segment. Dec 31 12:06:41.876645719 (620h37m41.747728754s) [noise=true , address=145.79.227.22:4511 , piece_id=42KZWHQXNTCCDHZWPXSI24QNB5RGHQTIPJLO27VYJAZ6LMCU326A, seg_pos={0 0}, upload=543845]: Uploading piece... Dec 31 12:06:41.876677730 (620h37m41.747760765s) [noise=true , address=77.90.31.12:4590 , piece_id=WL5GDUYDDZASJAJAKB3DORLUWDZMFFHH5ON3C7FLCX2FVAMWV6RA, seg_pos={0 0}, upload=543845]: Uploading piece... Dec 31 12:06:41.877729894 (620h37m41.748812929s) [noise=true , address=77.90.31.13:4626 , piece_id=ZOC7H557AUX34MJPWZK3Z4SZEIEKSEEV3U7YVBBXZEFFLKFACO7Q, seg_pos={0 0}, upload=543845]: Uploading piece... Dec 31 12:06:41.877835555 (620h37m41.7489186s) [noise=true , address=145.79.227.13:4803 , piece_id=CB2GGGL3WNQYMXM2F4TYAGQGYYGEEF2RDYRH6QGONVNKTL7RUWUA, seg_pos={0 0}, upload=543845]: Uploading piece... Dec 31 12:06:41.878129985 (620h37m41.74921303s) [noise=true , address=145.79.227.10:4403 , piece_id=6LYZBESPFPGMYEQ2M4UK2POFKXGKNZECSYKOKFIAD7TD7UN76RHQ, seg_pos={0 0}, upload=543845]: Uploading piece... Dec 31 12:06:41.879352833 (620h37m41.750435868s) [noise=true , address=145.79.227.33:4870 , piece_id=S3YQQ5HN3GT6PNPI6KSJCNBBGXRPKP3PWWEHS7T7VJTJ5E3TI6KA, seg_pos={0 0}, upload=543845]: Uploading piece... Dec 31 12:06:41.879680488 (620h37m41.750763523s) [noise=true , address=145.79.227.20:4438 , piece_id=W3VI6GT2AIGCCAWQ7JVROB4JY5ZR46MV3GIO4NP5QKOJIMMI6BPA, seg_pos={0 0}, upload=543845]: Uploading piece... Dec 31 12:06:41.880542810 (620h37m41.751625845s) [noise=true , address=145.79.227.48:4302 , piece_id=GIKNYCN2LIPOYZVQZPG55TNEQEQCEE3CTHIWVMOGGK3NF5AXCC5A, seg_pos={0 0}, upload=543845]: Uploading piece... Dec 31 12:06:41.882134881 (620h37m41.753217916s) [noise=true , address=45.91.116.5:4298 , piece_id=QCVT3PWTSHOR65KU5KIQ5A57M3R4VWVC6RRJAB74NWMJ3X25PAKQ, seg_pos={0 0}, upload=543845]: Uploading piece... Dec 31 12:06:41.883322933 (620h37m41.754405968s) [noise=true , address=145.79.227.38:4978 , piece_id=JJL3V2C24SBSQAJXZDJCPRYUNRJSQAYIWXQWNWQ5DM6UVSTN6TBQ, seg_pos={0 0}, upload=543845]: Uploading piece... Dec 31 12:06:41.883773491 (620h37m41.754856536s) [noise=true , address=45.91.116.13:4153 , piece_id=5MS7UZBLFDSPNM7KGBAZ3JBV3TLTKGTGHTJIMPBVB5ZAYTCLAGCQ, seg_pos={0 0}, upload=543845]: Uploading piece... Dec 31 12:06:41.884245090 (620h37m41.755328135s) [noise=true , address=145.79.227.32:4835 , piece_id=7LHKQJTZNCNF5OZNPLEKY3C22EXZEJTA3J2KDVTDQT5IKVFBMBGQ, seg_pos={0 0}, upload=543845]: Uploading piece... Dec 31 12:06:41.885279038 (620h37m41.756362073s) [noise=true , address=145.79.227.31:4797 , piece_id=SLGLUN6H6PFW77BJT5PJWEKUPTRSYYZYUSRFPH2SUNWPOLCMDXOA, seg_pos={0 0}, upload=543845]: Uploading piece... Dec 31 12:06:41.885618514 (620h37m41.756701549s) [noise=true , address=45.91.116.11:4117 , piece_id=TQJHCTNISD77D5QYGUZZ7H3QHIKHEHXPR3DUZYVMRFSOAJJEU2TQ, seg_pos={0 0}, upload=543845]: Uploading piece... Dec 31 12:06:41.886062761 (620h37m41.757145796s) [noise=true , address=145.79.227.45:4231 , piece_id=TQ6RA7EXIAZM34XMV5BITZZIXV3DOMYVVJWGVHLIBTHWZ3SPTLFA, seg_pos={0 0}, upload=543845]: Uploading piece... Dec 31 12:06:41.886535711 (620h37m41.757618746s) [noise=true , address=145.79.227.52:4446 , piece_id=Z4XLQ3JLGJGAAZRF4RHDQII7JZGGNSKY4Y3ZAFEFQTYTFYLFGBVA, seg_pos={0 0}, upload=543845]: Uploading piece... Dec 31 12:06:43.490334895 (620h37m43.36141794s) [upload=543845]: Next returned nil segment Dec 31 12:06:43.490350115 (620h37m43.36143316s) [upload=543845]: Waiting for error group managing segments... Dec 31 12:06:43.863952818 (620h37m43.735035854s) [noise=true , address=145.79.227.38:4978 , piece_id=JJL3V2C24SBSQAJXZDJCPRYUNRJSQAYIWXQWNWQ5DM6UVSTN6TBQ, seg_pos={0 0}, upload=543845]: Done uploading piece. err: <nil> Dec 31 12:06:43.873988027 (620h37m43.745071062s) [noise=true , address=145.79.227.32:4835 , piece_id=7LHKQJTZNCNF5OZNPLEKY3C22EXZEJTA3J2KDVTDQT5IKVFBMBGQ, seg_pos={0 0}, upload=543845]: Done uploading piece. err: <nil> Dec 31 12:06:43.874631562 (620h37m43.745714597s) [noise=true , address=145.79.227.13:4803 , piece_id=CB2GGGL3WNQYMXM2F4TYAGQGYYGEEF2RDYRH6QGONVNKTL7RUWUA, seg_pos={0 0}, upload=543845]: Done uploading piece. err: <nil> Dec 31 12:06:43.879999253 (620h37m43.751082298s) [noise=true , address=145.79.227.45:4231 , piece_id=TQ6RA7EXIAZM34XMV5BITZZIXV3DOMYVVJWGVHLIBTHWZ3SPTLFA, seg_pos={0 0}, upload=543845]: Done uploading piece. err: <nil> Dec 31 12:06:43.883538031 (620h37m43.754621077s) [noise=true , address=145.79.227.52:4446 , piece_id=Z4XLQ3JLGJGAAZRF4RHDQII7JZGGNSKY4Y3ZAFEFQTYTFYLFGBVA, seg_pos={0 0}, upload=543845]: Done uploading piece. err: <nil> Dec 31 12:06:43.888290691 (620h37m43.759373736s) [noise=true , address=145.79.227.10:4403 , piece_id=6LYZBESPFPGMYEQ2M4UK2POFKXGKNZECSYKOKFIAD7TD7UN76RHQ, seg_pos={0 0}, upload=543845]: Done uploading piece. err: <nil> Dec 31 12:06:43.955479686 (620h37m43.826562721s) [noise=true , address=145.79.227.31:4797 , piece_id=SLGLUN6H6PFW77BJT5PJWEKUPTRSYYZYUSRFPH2SUNWPOLCMDXOA, seg_pos={0 0}, upload=543845]: Done uploading piece. err: <nil> Dec 31 12:06:43.959556689 (620h37m43.830639724s) [noise=true , address=145.79.227.22:4511 , piece_id=42KZWHQXNTCCDHZWPXSI24QNB5RGHQTIPJLO27VYJAZ6LMCU326A, seg_pos={0 0}, upload=543845]: Done uploading piece. err: <nil> Dec 31 12:06:43.960814374 (620h37m43.831897409s) [noise=true , address=145.79.227.20:4438 , piece_id=W3VI6GT2AIGCCAWQ7JVROB4JY5ZR46MV3GIO4NP5QKOJIMMI6BPA, seg_pos={0 0}, upload=543845]: Done uploading piece. err: <nil> Dec 31 12:06:43.971489370 (620h37m43.842572406s) [noise=true , address=145.79.227.48:4302 , piece_id=GIKNYCN2LIPOYZVQZPG55TNEQEQCEE3CTHIWVMOGGK3NF5AXCC5A, seg_pos={0 0}, upload=543845]: Done uploading piece. err: <nil> Dec 31 12:06:44.056791706 (620h37m43.927874751s) [noise=true , address=77.90.31.12:4590 , piece_id=WL5GDUYDDZASJAJAKB3DORLUWDZMFFHH5ON3C7FLCX2FVAMWV6RA, seg_pos={0 0}, upload=543845]: Done uploading piece. err: <nil> Dec 31 12:06:44.205033196 (620h37m44.076116251s) [noise=true , address=45.91.116.13:4153 , piece_id=5MS7UZBLFDSPNM7KGBAZ3JBV3TLTKGTGHTJIMPBVB5ZAYTCLAGCQ, seg_pos={0 0}, upload=543845]: Done uploading piece. err: <nil> Dec 31 12:06:44.213267215 (620h37m44.08435026s) [noise=true , address=45.91.116.11:4117 , piece_id=TQJHCTNISD77D5QYGUZZ7H3QHIKHEHXPR3DUZYVMRFSOAJJEU2TQ, seg_pos={0 0}, upload=543845]: Done uploading piece. err: <nil> Dec 31 12:06:44.213356544 (620h37m44.084439589s) [seg_pos={0 0}, upload=543845]: Segment reached optimal threshold of 13 pieces. Dec 31 12:06:44.213622221 (620h37m44.084705256s) [noise=true , address=77.90.31.13:4626 , piece_id=ZOC7H557AUX34MJPWZK3Z4SZEIEKSEEV3U7YVBBXZEFFLKFACO7Q, seg_pos={0 0}, upload=543845]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: expected piece hash; context canceled; EOF Dec 31 12:06:44.213630516 (620h37m44.084713551s) [noise=true , address=77.90.31.13:4626 , piece_id=ZOC7H557AUX34MJPWZK3Z4SZEIEKSEEV3U7YVBBXZEFFLKFACO7Q, seg_pos={0 0}, upload=543845]: Long tail cancellation due to optimal threshold Dec 31 12:06:44.213661004 (620h37m44.084744059s) [noise=true , address=145.79.227.33:4870 , piece_id=S3YQQ5HN3GT6PNPI6KSJCNBBGXRPKP3PWWEHS7T7VJTJ5E3TI6KA, seg_pos={0 0}, upload=543845]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Dec 31 12:06:44.213671544 (620h37m44.084754589s) [noise=true , address=145.79.227.33:4870 , piece_id=S3YQQ5HN3GT6PNPI6KSJCNBBGXRPKP3PWWEHS7T7VJTJ5E3TI6KA, seg_pos={0 0}, upload=543845]: Long tail cancellation due to optimal threshold Dec 31 12:06:44.213934625 (620h37m44.085017671s) [noise=true , address=45.91.116.5:4298 , piece_id=QCVT3PWTSHOR65KU5KIQ5A57M3R4VWVC6RRJAB74NWMJ3X25PAKQ, seg_pos={0 0}, upload=543845]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 31 12:06:44.213943893 (620h37m44.085026938s) [noise=true , address=45.91.116.5:4298 , piece_id=QCVT3PWTSHOR65KU5KIQ5A57M3R4VWVC6RRJAB74NWMJ3X25PAKQ, seg_pos={0 0}, upload=543845]: Long tail cancellation due to optimal threshold Dec 31 12:06:44.214005160 (620h37m44.085088205s) [seg_pos={0 0}, upload=543845]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 12:06:44.214100481 (620h37m44.085183527s) [upload=543845]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 12:06:44.214108987 (620h37m44.085192043s) [upload=543845]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 12:06:44.512491223 (620h37m44.383574268s) [upload=543845]: Done uploading segments... Dec 31 12:06:44.512516701 (620h37m44.383599756s) [upload=543845]: Upload finished. err: <nil> Duration: 30.002844404s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [...X................X..........X.X......X.............X.....] duration: 5.001269912s trace: https://jaeger.dev.storj.io/trace/2c96062d1b5b5a44 FAIL (2025-12-31 12:07:09 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001296903s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................................X...X...................] duration: 10.022966278s trace: https://jaeger.dev.storj.io/trace/2e598bd65e5b1d02 FAIL (2025-12-31 12:06:57 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 10.022991746s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.X..............................................X...........] duration: 27.229745476s trace: https://jaeger.dev.storj.io/trace/3696061a2156809c FAIL (2025-12-31 12:06:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== .17:4794 , piece_id=ZCNDE7ID7HX43BDZ7KFQKZ6R5V5KUPZWCIRRUQNK65I73JMZQ2JQ, seg_pos={0 0}, upload=543695]: Uploading piece... Dec 31 11:56:12.262721887 (620h27m12.133804932s) [noise=true , address=193.149.164.143:4695 , piece_id=5CJUMEUXTJVQOCSHQKZD4ZHQBTFSMUPIQCI7QMJX2URBSUAIGIFA, seg_pos={0 0}, upload=543695]: Uploading piece... Dec 31 11:56:12.263130455 (620h27m12.1342135s) [noise=true , address=45.91.116.5:4326 , piece_id=VDLNHILWQD6ZTD7JMHQY5BDH2A4IHSJTVHJGDI4IK3OBDZSLOADA, seg_pos={0 0}, upload=543695]: Uploading piece... Dec 31 11:56:12.264175605 (620h27m12.13525865s) [noise=true , address=77.90.31.18:4812 , piece_id=KOGGG55EKOFTTT7OOZ5AULZHU2POHF5DMMSX2RM5M65FC3Y3AXRQ, seg_pos={0 0}, upload=543695]: Uploading piece... Dec 31 11:56:12.264193529 (620h27m12.135276574s) [noise=true , address=145.79.227.49:4358 , piece_id=RP2Y6L6CZD3XOL7XOU3HRDSYZZ6AENXT7LM6XVHZPIZAHSL26BPQ, seg_pos={0 0}, upload=543695]: Uploading piece... Dec 31 11:56:12.264609000 (620h27m12.135692035s) [noise=true , address=45.91.116.11:4145 , piece_id=ZX5FCIOENG6476SS6RDITOCPPCDBXAYOHAOAP4Q545FEBKSSQDDA, seg_pos={0 0}, upload=543695]: Uploading piece... Dec 31 11:56:12.264666079 (620h27m12.135749114s) [noise=true , address=64.49.13.140:4505 , piece_id=YRQ2VLRASSQVB65ONBLZXVLXY7COOSYVUUNMKJO5AOAJKFFIEW2A, seg_pos={0 0}, upload=543695]: Uploading piece... Dec 31 11:56:12.265310566 (620h27m12.136393611s) [noise=true , address=145.79.227.44:4206 , piece_id=TR747XVZACWUD6WGRSROIO5OHFLVBM3XT3XTEWA74GXNXMMYMQHQ, seg_pos={0 0}, upload=543695]: Uploading piece... Dec 31 11:56:12.265745504 (620h27m12.136828549s) [noise=true , address=145.79.227.33:4894 , piece_id=TFZVFOU3GA45OWXOWVA4R2A3W47CS6DCK5K4DAMYYMIH4J5CD23A, seg_pos={0 0}, upload=543695]: Uploading piece... Dec 31 11:56:12.266321861 (620h27m12.137404896s) [noise=true , address=45.91.116.7:4013 , piece_id=A4VDKYMR74JC325EFEP22V3TOGZRMTQB4UOE5I2Y7JKQF32AIZAA, seg_pos={0 0}, upload=543695]: Uploading piece... Dec 31 11:56:12.266868933 (620h27m12.137951978s) [noise=true , address=64.49.13.169:4707 , piece_id=ZQTZNZZQFGHXU2DIVBDUMJTYRAWAPRATLOGQ6AY4A6N45276NDXQ, seg_pos={0 0}, upload=543695]: Uploading piece... Dec 31 11:56:12.267258635 (620h27m12.13834167s) [noise=true , address=77.90.31.13:4657 , piece_id=KSNYKBDG4YOLLPTKC4OFSQR6XOBOGNA64PC3BCKQLRLYGCWFCUUQ, seg_pos={0 0}, upload=543695]: Uploading piece... Dec 31 11:56:12.267587522 (620h27m12.138670567s) [noise=true , address=145.79.227.40:4058 , piece_id=F3RLQOLC5C6JRKLBZICBMGRSJENDRTTP3PGJEFAOBXNHZP4QI3IA, seg_pos={0 0}, upload=543695]: Uploading piece... Dec 31 11:56:12.268440155 (620h27m12.13952319s) [noise=true , address=145.79.227.34:4939 , piece_id=TSCIX7EBMZ72GAOPGWHMCUW4PVEONBZ56A3ODVMIMY6LZZW2XDAQ, seg_pos={0 0}, upload=543695]: Uploading piece... Dec 31 11:56:12.269297889 (620h27m12.140380924s) [noise=true , address=145.79.227.2:4134 , piece_id=HWWXLVVLSMPGUKIP5A5T5IXAB3ZCABH7OA5DIJSNDXRQU23EIFNA, seg_pos={0 0}, upload=543695]: Uploading piece... Dec 31 11:56:13.698835153 (620h27m13.569918188s) [upload=543695]: Next returned nil segment Dec 31 11:56:13.698965801 (620h27m13.570048846s) [upload=543695]: Waiting for error group managing segments... Dec 31 11:56:14.047217086 (620h27m13.918300131s) [noise=true , address=64.49.13.140:4505 , piece_id=YRQ2VLRASSQVB65ONBLZXVLXY7COOSYVUUNMKJO5AOAJKFFIEW2A, seg_pos={0 0}, upload=543695]: Done uploading piece. err: <nil> Dec 31 11:56:14.137538488 (620h27m14.008621523s) [noise=true , address=145.79.227.2:4134 , piece_id=HWWXLVVLSMPGUKIP5A5T5IXAB3ZCABH7OA5DIJSNDXRQU23EIFNA, seg_pos={0 0}, upload=543695]: Done uploading piece. err: <nil> Dec 31 11:56:14.139917508 (620h27m14.011000543s) [noise=true , address=145.79.227.46:4619 , piece_id=BV3NLVQZM457CG2ZYJYZANIDKDVYCQWC64FZGRXFFDV6FFLC2SJA, seg_pos={0 0}, upload=543695]: Done uploading piece. err: <nil> Dec 31 11:56:14.472160101 (620h27m14.343243136s) [noise=true , address=77.90.31.17:4794 , piece_id=ZCNDE7ID7HX43BDZ7KFQKZ6R5V5KUPZWCIRRUQNK65I73JMZQ2JQ, seg_pos={0 0}, upload=543695]: Done uploading piece. err: <nil> Dec 31 11:56:14.476987874 (620h27m14.348070919s) [noise=true , address=45.91.116.5:4326 , piece_id=VDLNHILWQD6ZTD7JMHQY5BDH2A4IHSJTVHJGDI4IK3OBDZSLOADA, seg_pos={0 0}, upload=543695]: Done uploading piece. err: <nil> Dec 31 11:56:14.518004714 (620h27m14.389087749s) [noise=true , address=45.91.116.7:4013 , piece_id=A4VDKYMR74JC325EFEP22V3TOGZRMTQB4UOE5I2Y7JKQF32AIZAA, seg_pos={0 0}, upload=543695]: Done uploading piece. err: <nil> Dec 31 11:56:14.540552146 (620h27m14.411635191s) [noise=true , address=77.90.31.18:4812 , piece_id=KOGGG55EKOFTTT7OOZ5AULZHU2POHF5DMMSX2RM5M65FC3Y3AXRQ, seg_pos={0 0}, upload=543695]: Done uploading piece. err: <nil> Dec 31 11:56:14.677369210 (620h27m14.548452255s) [noise=true , address=145.79.227.40:4058 , piece_id=F3RLQOLC5C6JRKLBZICBMGRSJENDRTTP3PGJEFAOBXNHZP4QI3IA, seg_pos={0 0}, upload=543695]: Done uploading piece. err: <nil> Dec 31 11:56:15.034375707 (620h27m14.905458742s) [noise=true , address=145.79.227.44:4206 , piece_id=TR747XVZACWUD6WGRSROIO5OHFLVBM3XT3XTEWA74GXNXMMYMQHQ, seg_pos={0 0}, upload=543695]: Done uploading piece. err: <nil> Dec 31 11:56:15.095619932 (620h27m14.966702967s) [noise=true , address=145.79.227.49:4358 , piece_id=RP2Y6L6CZD3XOL7XOU3HRDSYZZ6AENXT7LM6XVHZPIZAHSL26BPQ, seg_pos={0 0}, upload=543695]: Done uploading piece. err: <nil> Dec 31 11:56:15.099851950 (620h27m14.970934995s) [noise=true , address=145.79.227.43:4164 , piece_id=QQ4D2S4XAMIDBDSZNYM2G5WT4V7DR4E5WHDSYVT4SZ4YHANAY4QA, seg_pos={0 0}, upload=543695]: Done uploading piece. err: <nil> Dec 31 11:56:15.113070049 (620h27m14.984153094s) [noise=true , address=145.79.227.33:4894 , piece_id=TFZVFOU3GA45OWXOWVA4R2A3W47CS6DCK5K4DAMYYMIH4J5CD23A, seg_pos={0 0}, upload=543695]: Done uploading piece. err: <nil> Dec 31 11:56:15.127931066 (620h27m14.999014101s) [noise=true , address=145.79.227.30:4792 , piece_id=B72NRZMLRWZQRV222OBU6EPXZYYF4K6PXT6L77XZKNVZOA25NLIQ, seg_pos={0 0}, upload=543695]: Done uploading piece. err: <nil> Dec 31 11:56:15.147826669 (620h27m15.018909714s) [noise=true , address=193.149.164.143:4695 , piece_id=5CJUMEUXTJVQOCSHQKZD4ZHQBTFSMUPIQCI7QMJX2URBSUAIGIFA, seg_pos={0 0}, upload=543695]: Done uploading piece. err: <nil> Dec 31 11:56:15.147845695 (620h27m15.01892874s) [seg_pos={0 0}, upload=543695]: Segment reached optimal threshold of 14 pieces. Dec 31 11:56:15.148175553 (620h27m15.019258598s) [noise=true , address=145.79.227.34:4939 , piece_id=TSCIX7EBMZ72GAOPGWHMCUW4PVEONBZ56A3ODVMIMY6LZZW2XDAQ, seg_pos={0 0}, upload=543695]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1h7xrPMs5kHtjm7hiB4hfDaZUHFgtJNoMGqdy3HFif2Aned8eu): protocol: expected piece hash; context canceled; EOF Dec 31 11:56:15.148194439 (620h27m15.019277484s) [noise=true , address=145.79.227.34:4939 , piece_id=TSCIX7EBMZ72GAOPGWHMCUW4PVEONBZ56A3ODVMIMY6LZZW2XDAQ, seg_pos={0 0}, upload=543695]: Long tail cancellation due to optimal threshold Dec 31 11:56:15.148491545 (620h27m15.01957458s) [noise=true , address=77.90.31.13:4657 , piece_id=KSNYKBDG4YOLLPTKC4OFSQR6XOBOGNA64PC3BCKQLRLYGCWFCUUQ, seg_pos={0 0}, upload=543695]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1LLSnWuMa6W7y2jhFUqXoSPEsTyrFsKi8FQDNXF8Tv7XHbP6yt): protocol: write tcp 188.166.225.170:49614->77.90.31.13:4657: use of closed network connection; write tcp 188.166.225.170:49614->77.90.31.13:4657: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49614->77.90.31.13:4657: use of closed network connection Dec 31 11:56:15.148498648 (620h27m15.019581693s) [noise=true , address=77.90.31.13:4657 , piece_id=KSNYKBDG4YOLLPTKC4OFSQR6XOBOGNA64PC3BCKQLRLYGCWFCUUQ, seg_pos={0 0}, upload=543695]: Long tail cancellation due to optimal threshold Dec 31 11:56:15.148641420 (620h27m15.019724465s) [noise=true , address=45.91.116.11:4145 , piece_id=ZX5FCIOENG6476SS6RDITOCPPCDBXAYOHAOAP4Q545FEBKSSQDDA, seg_pos={0 0}, upload=543695]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WKV8PJxq5Kyb3ubCbrfbyYaX6fvRkPMKW2ssUteMFNHm9fP4E): protocol: expected piece hash; context canceled; EOF Dec 31 11:56:15.148647441 (620h27m15.019730476s) [noise=true , address=45.91.116.11:4145 , piece_id=ZX5FCIOENG6476SS6RDITOCPPCDBXAYOHAOAP4Q545FEBKSSQDDA, seg_pos={0 0}, upload=543695]: Long tail cancellation due to optimal threshold Dec 31 11:56:15.148750738 (620h27m15.019833773s) [noise=true , address=64.49.13.169:4707 , piece_id=ZQTZNZZQFGHXU2DIVBDUMJTYRAWAPRATLOGQ6AY4A6N45276NDXQ, seg_pos={0 0}, upload=543695]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:45002->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:45002->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45002->64.49.13.169:4707: use of closed network connection Dec 31 11:56:15.148766589 (620h27m15.019849634s) [noise=true , address=64.49.13.169:4707 , piece_id=ZQTZNZZQFGHXU2DIVBDUMJTYRAWAPRATLOGQ6AY4A6N45276NDXQ, seg_pos={0 0}, upload=543695]: Long tail cancellation due to optimal threshold Dec 31 11:56:15.148782148 (620h27m15.019865193s) [seg_pos={0 0}, upload=543695]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 11:56:15.148845759 (620h27m15.019928804s) [upload=543695]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 11:56:15.148850428 (620h27m15.019933463s) [upload=543695]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 11:56:15.442319734 (620h27m15.313402779s) [upload=543695]: Done uploading segments... Dec 31 11:56:15.442337046 (620h27m15.313420081s) [upload=543695]: Upload finished. err: <nil> Duration: 27.229759334s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [....X................X..........X.X......X.............X....] duration: 3.375499463s trace: https://jaeger.dev.storj.io/trace/2c96062d1b5b5a44 FAIL (2025-12-31 12:06:44 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.375525323s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..X..............................................X..........] duration: 27.917589693s trace: https://jaeger.dev.storj.io/trace/3696061a2156809c FAIL (2025-12-31 12:06:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== .17:4794 , piece_id=ZCNDE7ID7HX43BDZ7KFQKZ6R5V5KUPZWCIRRUQNK65I73JMZQ2JQ, seg_pos={0 0}, upload=543695]: Uploading piece... Dec 31 11:56:12.262721887 (620h27m12.133804932s) [noise=true , address=193.149.164.143:4695 , piece_id=5CJUMEUXTJVQOCSHQKZD4ZHQBTFSMUPIQCI7QMJX2URBSUAIGIFA, seg_pos={0 0}, upload=543695]: Uploading piece... Dec 31 11:56:12.263130455 (620h27m12.1342135s) [noise=true , address=45.91.116.5:4326 , piece_id=VDLNHILWQD6ZTD7JMHQY5BDH2A4IHSJTVHJGDI4IK3OBDZSLOADA, seg_pos={0 0}, upload=543695]: Uploading piece... Dec 31 11:56:12.264175605 (620h27m12.13525865s) [noise=true , address=77.90.31.18:4812 , piece_id=KOGGG55EKOFTTT7OOZ5AULZHU2POHF5DMMSX2RM5M65FC3Y3AXRQ, seg_pos={0 0}, upload=543695]: Uploading piece... Dec 31 11:56:12.264193529 (620h27m12.135276574s) [noise=true , address=145.79.227.49:4358 , piece_id=RP2Y6L6CZD3XOL7XOU3HRDSYZZ6AENXT7LM6XVHZPIZAHSL26BPQ, seg_pos={0 0}, upload=543695]: Uploading piece... Dec 31 11:56:12.264609000 (620h27m12.135692035s) [noise=true , address=45.91.116.11:4145 , piece_id=ZX5FCIOENG6476SS6RDITOCPPCDBXAYOHAOAP4Q545FEBKSSQDDA, seg_pos={0 0}, upload=543695]: Uploading piece... Dec 31 11:56:12.264666079 (620h27m12.135749114s) [noise=true , address=64.49.13.140:4505 , piece_id=YRQ2VLRASSQVB65ONBLZXVLXY7COOSYVUUNMKJO5AOAJKFFIEW2A, seg_pos={0 0}, upload=543695]: Uploading piece... Dec 31 11:56:12.265310566 (620h27m12.136393611s) [noise=true , address=145.79.227.44:4206 , piece_id=TR747XVZACWUD6WGRSROIO5OHFLVBM3XT3XTEWA74GXNXMMYMQHQ, seg_pos={0 0}, upload=543695]: Uploading piece... Dec 31 11:56:12.265745504 (620h27m12.136828549s) [noise=true , address=145.79.227.33:4894 , piece_id=TFZVFOU3GA45OWXOWVA4R2A3W47CS6DCK5K4DAMYYMIH4J5CD23A, seg_pos={0 0}, upload=543695]: Uploading piece... Dec 31 11:56:12.266321861 (620h27m12.137404896s) [noise=true , address=45.91.116.7:4013 , piece_id=A4VDKYMR74JC325EFEP22V3TOGZRMTQB4UOE5I2Y7JKQF32AIZAA, seg_pos={0 0}, upload=543695]: Uploading piece... Dec 31 11:56:12.266868933 (620h27m12.137951978s) [noise=true , address=64.49.13.169:4707 , piece_id=ZQTZNZZQFGHXU2DIVBDUMJTYRAWAPRATLOGQ6AY4A6N45276NDXQ, seg_pos={0 0}, upload=543695]: Uploading piece... Dec 31 11:56:12.267258635 (620h27m12.13834167s) [noise=true , address=77.90.31.13:4657 , piece_id=KSNYKBDG4YOLLPTKC4OFSQR6XOBOGNA64PC3BCKQLRLYGCWFCUUQ, seg_pos={0 0}, upload=543695]: Uploading piece... Dec 31 11:56:12.267587522 (620h27m12.138670567s) [noise=true , address=145.79.227.40:4058 , piece_id=F3RLQOLC5C6JRKLBZICBMGRSJENDRTTP3PGJEFAOBXNHZP4QI3IA, seg_pos={0 0}, upload=543695]: Uploading piece... Dec 31 11:56:12.268440155 (620h27m12.13952319s) [noise=true , address=145.79.227.34:4939 , piece_id=TSCIX7EBMZ72GAOPGWHMCUW4PVEONBZ56A3ODVMIMY6LZZW2XDAQ, seg_pos={0 0}, upload=543695]: Uploading piece... Dec 31 11:56:12.269297889 (620h27m12.140380924s) [noise=true , address=145.79.227.2:4134 , piece_id=HWWXLVVLSMPGUKIP5A5T5IXAB3ZCABH7OA5DIJSNDXRQU23EIFNA, seg_pos={0 0}, upload=543695]: Uploading piece... Dec 31 11:56:13.698835153 (620h27m13.569918188s) [upload=543695]: Next returned nil segment Dec 31 11:56:13.698965801 (620h27m13.570048846s) [upload=543695]: Waiting for error group managing segments... Dec 31 11:56:14.047217086 (620h27m13.918300131s) [noise=true , address=64.49.13.140:4505 , piece_id=YRQ2VLRASSQVB65ONBLZXVLXY7COOSYVUUNMKJO5AOAJKFFIEW2A, seg_pos={0 0}, upload=543695]: Done uploading piece. err: <nil> Dec 31 11:56:14.137538488 (620h27m14.008621523s) [noise=true , address=145.79.227.2:4134 , piece_id=HWWXLVVLSMPGUKIP5A5T5IXAB3ZCABH7OA5DIJSNDXRQU23EIFNA, seg_pos={0 0}, upload=543695]: Done uploading piece. err: <nil> Dec 31 11:56:14.139917508 (620h27m14.011000543s) [noise=true , address=145.79.227.46:4619 , piece_id=BV3NLVQZM457CG2ZYJYZANIDKDVYCQWC64FZGRXFFDV6FFLC2SJA, seg_pos={0 0}, upload=543695]: Done uploading piece. err: <nil> Dec 31 11:56:14.472160101 (620h27m14.343243136s) [noise=true , address=77.90.31.17:4794 , piece_id=ZCNDE7ID7HX43BDZ7KFQKZ6R5V5KUPZWCIRRUQNK65I73JMZQ2JQ, seg_pos={0 0}, upload=543695]: Done uploading piece. err: <nil> Dec 31 11:56:14.476987874 (620h27m14.348070919s) [noise=true , address=45.91.116.5:4326 , piece_id=VDLNHILWQD6ZTD7JMHQY5BDH2A4IHSJTVHJGDI4IK3OBDZSLOADA, seg_pos={0 0}, upload=543695]: Done uploading piece. err: <nil> Dec 31 11:56:14.518004714 (620h27m14.389087749s) [noise=true , address=45.91.116.7:4013 , piece_id=A4VDKYMR74JC325EFEP22V3TOGZRMTQB4UOE5I2Y7JKQF32AIZAA, seg_pos={0 0}, upload=543695]: Done uploading piece. err: <nil> Dec 31 11:56:14.540552146 (620h27m14.411635191s) [noise=true , address=77.90.31.18:4812 , piece_id=KOGGG55EKOFTTT7OOZ5AULZHU2POHF5DMMSX2RM5M65FC3Y3AXRQ, seg_pos={0 0}, upload=543695]: Done uploading piece. err: <nil> Dec 31 11:56:14.677369210 (620h27m14.548452255s) [noise=true , address=145.79.227.40:4058 , piece_id=F3RLQOLC5C6JRKLBZICBMGRSJENDRTTP3PGJEFAOBXNHZP4QI3IA, seg_pos={0 0}, upload=543695]: Done uploading piece. err: <nil> Dec 31 11:56:15.034375707 (620h27m14.905458742s) [noise=true , address=145.79.227.44:4206 , piece_id=TR747XVZACWUD6WGRSROIO5OHFLVBM3XT3XTEWA74GXNXMMYMQHQ, seg_pos={0 0}, upload=543695]: Done uploading piece. err: <nil> Dec 31 11:56:15.095619932 (620h27m14.966702967s) [noise=true , address=145.79.227.49:4358 , piece_id=RP2Y6L6CZD3XOL7XOU3HRDSYZZ6AENXT7LM6XVHZPIZAHSL26BPQ, seg_pos={0 0}, upload=543695]: Done uploading piece. err: <nil> Dec 31 11:56:15.099851950 (620h27m14.970934995s) [noise=true , address=145.79.227.43:4164 , piece_id=QQ4D2S4XAMIDBDSZNYM2G5WT4V7DR4E5WHDSYVT4SZ4YHANAY4QA, seg_pos={0 0}, upload=543695]: Done uploading piece. err: <nil> Dec 31 11:56:15.113070049 (620h27m14.984153094s) [noise=true , address=145.79.227.33:4894 , piece_id=TFZVFOU3GA45OWXOWVA4R2A3W47CS6DCK5K4DAMYYMIH4J5CD23A, seg_pos={0 0}, upload=543695]: Done uploading piece. err: <nil> Dec 31 11:56:15.127931066 (620h27m14.999014101s) [noise=true , address=145.79.227.30:4792 , piece_id=B72NRZMLRWZQRV222OBU6EPXZYYF4K6PXT6L77XZKNVZOA25NLIQ, seg_pos={0 0}, upload=543695]: Done uploading piece. err: <nil> Dec 31 11:56:15.147826669 (620h27m15.018909714s) [noise=true , address=193.149.164.143:4695 , piece_id=5CJUMEUXTJVQOCSHQKZD4ZHQBTFSMUPIQCI7QMJX2URBSUAIGIFA, seg_pos={0 0}, upload=543695]: Done uploading piece. err: <nil> Dec 31 11:56:15.147845695 (620h27m15.01892874s) [seg_pos={0 0}, upload=543695]: Segment reached optimal threshold of 14 pieces. Dec 31 11:56:15.148175553 (620h27m15.019258598s) [noise=true , address=145.79.227.34:4939 , piece_id=TSCIX7EBMZ72GAOPGWHMCUW4PVEONBZ56A3ODVMIMY6LZZW2XDAQ, seg_pos={0 0}, upload=543695]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1h7xrPMs5kHtjm7hiB4hfDaZUHFgtJNoMGqdy3HFif2Aned8eu): protocol: expected piece hash; context canceled; EOF Dec 31 11:56:15.148194439 (620h27m15.019277484s) [noise=true , address=145.79.227.34:4939 , piece_id=TSCIX7EBMZ72GAOPGWHMCUW4PVEONBZ56A3ODVMIMY6LZZW2XDAQ, seg_pos={0 0}, upload=543695]: Long tail cancellation due to optimal threshold Dec 31 11:56:15.148491545 (620h27m15.01957458s) [noise=true , address=77.90.31.13:4657 , piece_id=KSNYKBDG4YOLLPTKC4OFSQR6XOBOGNA64PC3BCKQLRLYGCWFCUUQ, seg_pos={0 0}, upload=543695]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1LLSnWuMa6W7y2jhFUqXoSPEsTyrFsKi8FQDNXF8Tv7XHbP6yt): protocol: write tcp 188.166.225.170:49614->77.90.31.13:4657: use of closed network connection; write tcp 188.166.225.170:49614->77.90.31.13:4657: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49614->77.90.31.13:4657: use of closed network connection Dec 31 11:56:15.148498648 (620h27m15.019581693s) [noise=true , address=77.90.31.13:4657 , piece_id=KSNYKBDG4YOLLPTKC4OFSQR6XOBOGNA64PC3BCKQLRLYGCWFCUUQ, seg_pos={0 0}, upload=543695]: Long tail cancellation due to optimal threshold Dec 31 11:56:15.148641420 (620h27m15.019724465s) [noise=true , address=45.91.116.11:4145 , piece_id=ZX5FCIOENG6476SS6RDITOCPPCDBXAYOHAOAP4Q545FEBKSSQDDA, seg_pos={0 0}, upload=543695]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WKV8PJxq5Kyb3ubCbrfbyYaX6fvRkPMKW2ssUteMFNHm9fP4E): protocol: expected piece hash; context canceled; EOF Dec 31 11:56:15.148647441 (620h27m15.019730476s) [noise=true , address=45.91.116.11:4145 , piece_id=ZX5FCIOENG6476SS6RDITOCPPCDBXAYOHAOAP4Q545FEBKSSQDDA, seg_pos={0 0}, upload=543695]: Long tail cancellation due to optimal threshold Dec 31 11:56:15.148750738 (620h27m15.019833773s) [noise=true , address=64.49.13.169:4707 , piece_id=ZQTZNZZQFGHXU2DIVBDUMJTYRAWAPRATLOGQ6AY4A6N45276NDXQ, seg_pos={0 0}, upload=543695]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:45002->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:45002->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45002->64.49.13.169:4707: use of closed network connection Dec 31 11:56:15.148766589 (620h27m15.019849634s) [noise=true , address=64.49.13.169:4707 , piece_id=ZQTZNZZQFGHXU2DIVBDUMJTYRAWAPRATLOGQ6AY4A6N45276NDXQ, seg_pos={0 0}, upload=543695]: Long tail cancellation due to optimal threshold Dec 31 11:56:15.148782148 (620h27m15.019865193s) [seg_pos={0 0}, upload=543695]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 11:56:15.148845759 (620h27m15.019928804s) [upload=543695]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 11:56:15.148850428 (620h27m15.019933463s) [upload=543695]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 11:56:15.442319734 (620h27m15.313402779s) [upload=543695]: Done uploading segments... Dec 31 11:56:15.442337046 (620h27m15.313420081s) [upload=543695]: Upload finished. err: <nil> Duration: 27.917617476s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [X....X................X..........X.X......X.............X...] duration: 3.802479886s trace: https://jaeger.dev.storj.io/trace/2c96062d1b5b5a44 FAIL (2025-12-31 12:06:01 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.802511667s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................................X...X..................] duration: 11.407911453s trace: https://jaeger.dev.storj.io/trace/2e598bd65e5b1d02 FAIL (2025-12-31 12:05:57 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 11.407937221s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [........X..X..XX....X....X.....XX.XX.X....XX.....X.....X....] duration: 6.023663845s trace: https://jaeger.dev.storj.io/trace/2aeaacdc7b2ec FAIL (2025-12-31 12:05:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 31 12:03:06.696854561 (620h34m6.567937596s) [upload=543793]: Starting upload Dec 31 12:03:06.696906339 (620h34m6.567989384s) [upload=543793]: Done starting upload Dec 31 12:03:06.697022900 (620h34m6.568105945s) [upload=543793]: Uploading segments... Dec 31 12:03:06.697101751 (620h34m6.568184786s) [upload=543793]: Got next segment. Inline: false Dec 31 12:03:06.697136497 (620h34m6.568219542s) [upload=543793]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 12:03:06.697142539 (620h34m6.568225584s) [upload=543793]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 12:03:06.941461997 (620h34m6.812545032s) [seg_pos={0 0}, upload=543793]: Begin upload segment... Dec 31 12:03:06.941527512 (620h34m6.812610557s) [seg_pos={0 0}, upload=543793]: Done begin upload segment. Dec 31 12:03:06.941595150 (620h34m6.812678185s) [noise=true , address=77.90.31.21:4878 , piece_id=6IFN4VOBLW2F3HQBWQC3VAHKU2P4EUQM232KRYBXVDSVQADA5YUA, seg_pos={0 0}, upload=543793]: Uploading piece... Dec 31 12:03:06.941627272 (620h34m6.812710317s) [noise=true , address=145.79.227.31:4798 , piece_id=WOUKZMWL6E77DEHNOFT65LE2JE77W7GFZ4U4IFTN6SEYVOQ4SBUA, seg_pos={0 0}, upload=543793]: Uploading piece... Dec 31 12:03:06.941722613 (620h34m6.812805649s) [noise=true , address=145.79.227.37:4566 , piece_id=C4LJ4JIQHTUH72K26O3YTU4CYPOO4XBH7EOMMFPIDB64JB3LZVTA, seg_pos={0 0}, upload=543793]: Uploading piece... Dec 31 12:03:06.942242333 (620h34m6.813325378s) [noise=true , address=145.79.227.8:4329 , piece_id=LXMQABMWVOOCIBKZ2EL7HTW5QTUJGB3FDTZAFELIM7DYZS42WNLQ, seg_pos={0 0}, upload=543793]: Uploading piece... Dec 31 12:03:06.942409090 (620h34m6.813492125s) [noise=true , address=45.91.116.5:4297 , piece_id=32QRF6F4NAVYHWBIOYTLKBD6RJOE3A5DO7N3THJAIPYSBADKLEBQ, seg_pos={0 0}, upload=543793]: Uploading piece... Dec 31 12:03:06.942600695 (620h34m6.81368373s) [noise=true , address=145.79.227.14:4837 , piece_id=LOEPZ4EIZKRWQYII7DUWRJHWBI7MCFHOLJLDZ5EWHPQBRAUXNCOQ, seg_pos={0 0}, upload=543793]: Uploading piece... Dec 31 12:03:06.942760429 (620h34m6.813843464s) [noise=true , address=145.79.227.45:4230 , piece_id=NTHVSF643LXDW5MQ5DQGM3BHPTIXOEL5NEKCLW6ZR7MIJLODF5LA, seg_pos={0 0}, upload=543793]: Uploading piece... Dec 31 12:03:06.943142737 (620h34m6.814225772s) [noise=true , address=145.79.227.36:4517 , piece_id=LGLPG7PMCKGSQ5TRKVQFHZB5DXNMPUKUHFV5AC7CFVACLIEIXQHA, seg_pos={0 0}, upload=543793]: Uploading piece... Dec 31 12:03:06.943205877 (620h34m6.814288912s) [noise=true , address=145.79.227.19:4078 , piece_id=K5G5LEONW3RAWGLW3JAZSK7QW6QFHROZO6PX3EF5A3AJKZAXKK6A, seg_pos={0 0}, upload=543793]: Uploading piece... Dec 31 12:03:06.943520025 (620h34m6.81460307s) [noise=true , address=145.79.227.50:4375 , piece_id=K22R6U4PNLO7ZNOBTKYVJ6URJX5CACIDPQ44DRQYQW4BADMHK4SQ, seg_pos={0 0}, upload=543793]: Uploading piece... Dec 31 12:03:06.943967728 (620h34m6.815050773s) [noise=true , address=145.79.227.13:4802 , piece_id=Y35WHD72DB5NBDPIOJVWKMZ2S7MZSKWUJZAIEKVACFML7SZBNKNQ, seg_pos={0 0}, upload=543793]: Uploading piece... Dec 31 12:03:06.944351178 (620h34m6.815434223s) [noise=true , address=77.90.31.16:4736 , piece_id=67UHO2QV3YFPWCDJJPSCHFDEUHFZU4O4YJCAK2KXYE2HMOKNQGZA, seg_pos={0 0}, upload=543793]: Uploading piece... Dec 31 12:03:06.944709330 (620h34m6.815792365s) [noise=true , address=145.79.227.22:4509 , piece_id=YC2PT7M7N7VG5FL4Q2BRDULJDY47HSKSNJ6KWYH5JZGWZTYBFOUA, seg_pos={0 0}, upload=543793]: Uploading piece... Dec 31 12:03:06.945098561 (620h34m6.816181596s) [noise=true , address=145.79.227.42:4121 , piece_id=2GLF77AV622O5PLEGOWPZR36PWHLD34W7EC24J242ZMPEMQPS7AQ, seg_pos={0 0}, upload=543793]: Uploading piece... Dec 31 12:03:06.945725214 (620h34m6.816808249s) [noise=true , address=45.91.116.6:4975 , piece_id=XMLZC7GCIQXV7YV32HKDBKBDODOVMBCORVKEV7SZC4POWLIQNMOQ, seg_pos={0 0}, upload=543793]: Uploading piece... Dec 31 12:03:06.947935113 (620h34m6.819018158s) [noise=true , address=77.90.31.11:4555 , piece_id=NN2EBC66KECKQIXTKB2ZVSM2YVLKLXCCLQWMP5RN2C5UWZM5KFFQ, seg_pos={0 0}, upload=543793]: Uploading piece... Dec 31 12:03:08.633787608 (620h34m8.504870663s) [upload=543793]: Next returned nil segment Dec 31 12:03:08.633836921 (620h34m8.504919967s) [upload=543793]: Waiting for error group managing segments... Dec 31 12:03:08.914501466 (620h34m8.785584501s) [noise=true , address=145.79.227.45:4230 , piece_id=NTHVSF643LXDW5MQ5DQGM3BHPTIXOEL5NEKCLW6ZR7MIJLODF5LA, seg_pos={0 0}, upload=543793]: Done uploading piece. err: <nil> Dec 31 12:03:08.919724370 (620h34m8.790807405s) [noise=true , address=145.79.227.22:4509 , piece_id=YC2PT7M7N7VG5FL4Q2BRDULJDY47HSKSNJ6KWYH5JZGWZTYBFOUA, seg_pos={0 0}, upload=543793]: Done uploading piece. err: <nil> Dec 31 12:03:08.927631316 (620h34m8.798714351s) [noise=true , address=145.79.227.14:4837 , piece_id=LOEPZ4EIZKRWQYII7DUWRJHWBI7MCFHOLJLDZ5EWHPQBRAUXNCOQ, seg_pos={0 0}, upload=543793]: Done uploading piece. err: <nil> Dec 31 12:03:08.932530494 (620h34m8.803613539s) [noise=true , address=145.79.227.36:4517 , piece_id=LGLPG7PMCKGSQ5TRKVQFHZB5DXNMPUKUHFV5AC7CFVACLIEIXQHA, seg_pos={0 0}, upload=543793]: Done uploading piece. err: <nil> Dec 31 12:03:08.937408452 (620h34m8.808491488s) [noise=true , address=145.79.227.37:4566 , piece_id=C4LJ4JIQHTUH72K26O3YTU4CYPOO4XBH7EOMMFPIDB64JB3LZVTA, seg_pos={0 0}, upload=543793]: Done uploading piece. err: <nil> Dec 31 12:03:08.959456975 (620h34m8.83054001s) [noise=true , address=145.79.227.31:4798 , piece_id=WOUKZMWL6E77DEHNOFT65LE2JE77W7GFZ4U4IFTN6SEYVOQ4SBUA, seg_pos={0 0}, upload=543793]: Done uploading piece. err: <nil> Dec 31 12:03:08.959912633 (620h34m8.830995688s) [noise=true , address=145.79.227.8:4329 , piece_id=LXMQABMWVOOCIBKZ2EL7HTW5QTUJGB3FDTZAFELIM7DYZS42WNLQ, seg_pos={0 0}, upload=543793]: Done uploading piece. err: <nil> Dec 31 12:03:08.960449174 (620h34m8.831532219s) [noise=true , address=145.79.227.42:4121 , piece_id=2GLF77AV622O5PLEGOWPZR36PWHLD34W7EC24J242ZMPEMQPS7AQ, seg_pos={0 0}, upload=543793]: Done uploading piece. err: <nil> Dec 31 12:03:08.961014460 (620h34m8.832097495s) [noise=true , address=145.79.227.50:4375 , piece_id=K22R6U4PNLO7ZNOBTKYVJ6URJX5CACIDPQ44DRQYQW4BADMHK4SQ, seg_pos={0 0}, upload=543793]: Done uploading piece. err: <nil> Dec 31 12:03:08.999315790 (620h34m8.870398825s) [noise=true , address=145.79.227.19:4078 , piece_id=K5G5LEONW3RAWGLW3JAZSK7QW6QFHROZO6PX3EF5A3AJKZAXKK6A, seg_pos={0 0}, upload=543793]: Done uploading piece. err: <nil> Dec 31 12:03:09.017297168 (620h34m8.888380204s) [noise=true , address=145.79.227.13:4802 , piece_id=Y35WHD72DB5NBDPIOJVWKMZ2S7MZSKWUJZAIEKVACFML7SZBNKNQ, seg_pos={0 0}, upload=543793]: Done uploading piece. err: <nil> Dec 31 12:03:09.181590309 (620h34m9.052673344s) [noise=true , address=77.90.31.21:4878 , piece_id=6IFN4VOBLW2F3HQBWQC3VAHKU2P4EUQM232KRYBXVDSVQADA5YUA, seg_pos={0 0}, upload=543793]: Done uploading piece. err: <nil> Dec 31 12:03:09.243812086 (620h34m9.114895121s) [noise=true , address=77.90.31.11:4555 , piece_id=NN2EBC66KECKQIXTKB2ZVSM2YVLKLXCCLQWMP5RN2C5UWZM5KFFQ, seg_pos={0 0}, upload=543793]: Done uploading piece. err: <nil> Dec 31 12:03:09.243837224 (620h34m9.114920269s) [seg_pos={0 0}, upload=543793]: Segment reached optimal threshold of 13 pieces. Dec 31 12:03:09.244070898 (620h34m9.115153943s) [noise=true , address=45.91.116.5:4297 , piece_id=32QRF6F4NAVYHWBIOYTLKBD6RJOE3A5DO7N3THJAIPYSBADKLEBQ, seg_pos={0 0}, upload=543793]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 31 12:03:09.244102789 (620h34m9.115185824s) [noise=true , address=45.91.116.5:4297 , piece_id=32QRF6F4NAVYHWBIOYTLKBD6RJOE3A5DO7N3THJAIPYSBADKLEBQ, seg_pos={0 0}, upload=543793]: Long tail cancellation due to optimal threshold Dec 31 12:03:09.244379997 (620h34m9.115463032s) [noise=true , address=77.90.31.16:4736 , piece_id=67UHO2QV3YFPWCDJJPSCHFDEUHFZU4O4YJCAK2KXYE2HMOKNQGZA, seg_pos={0 0}, upload=543793]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 31 12:03:09.244387681 (620h34m9.115470727s) [noise=true , address=77.90.31.16:4736 , piece_id=67UHO2QV3YFPWCDJJPSCHFDEUHFZU4O4YJCAK2KXYE2HMOKNQGZA, seg_pos={0 0}, upload=543793]: Long tail cancellation due to optimal threshold Dec 31 12:03:09.244536826 (620h34m9.115619861s) [noise=true , address=45.91.116.6:4975 , piece_id=XMLZC7GCIQXV7YV32HKDBKBDODOVMBCORVKEV7SZC4POWLIQNMOQ, seg_pos={0 0}, upload=543793]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 31 12:03:09.244545593 (620h34m9.115628638s) [noise=true , address=45.91.116.6:4975 , piece_id=XMLZC7GCIQXV7YV32HKDBKBDODOVMBCORVKEV7SZC4POWLIQNMOQ, seg_pos={0 0}, upload=543793]: Long tail cancellation due to optimal threshold Dec 31 12:03:09.244565681 (620h34m9.115648716s) [seg_pos={0 0}, upload=543793]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 12:03:09.244644411 (620h34m9.115727456s) [upload=543793]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 12:03:09.244650142 (620h34m9.115733187s) [upload=543793]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 12:03:09.534801749 (620h34m9.405884784s) [upload=543793]: Done uploading segments... Dec 31 12:03:09.534822359 (620h34m9.405905404s) [upload=543793]: Upload finished. err: <nil> Duration: 6.023692269s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...X..............................................X.........] duration: 30.391665837s trace: https://jaeger.dev.storj.io/trace/3696061a2156809c FAIL (2025-12-31 12:05:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== .17:4794 , piece_id=ZCNDE7ID7HX43BDZ7KFQKZ6R5V5KUPZWCIRRUQNK65I73JMZQ2JQ, seg_pos={0 0}, upload=543695]: Uploading piece... Dec 31 11:56:12.262721887 (620h27m12.133804932s) [noise=true , address=193.149.164.143:4695 , piece_id=5CJUMEUXTJVQOCSHQKZD4ZHQBTFSMUPIQCI7QMJX2URBSUAIGIFA, seg_pos={0 0}, upload=543695]: Uploading piece... Dec 31 11:56:12.263130455 (620h27m12.1342135s) [noise=true , address=45.91.116.5:4326 , piece_id=VDLNHILWQD6ZTD7JMHQY5BDH2A4IHSJTVHJGDI4IK3OBDZSLOADA, seg_pos={0 0}, upload=543695]: Uploading piece... Dec 31 11:56:12.264175605 (620h27m12.13525865s) [noise=true , address=77.90.31.18:4812 , piece_id=KOGGG55EKOFTTT7OOZ5AULZHU2POHF5DMMSX2RM5M65FC3Y3AXRQ, seg_pos={0 0}, upload=543695]: Uploading piece... Dec 31 11:56:12.264193529 (620h27m12.135276574s) [noise=true , address=145.79.227.49:4358 , piece_id=RP2Y6L6CZD3XOL7XOU3HRDSYZZ6AENXT7LM6XVHZPIZAHSL26BPQ, seg_pos={0 0}, upload=543695]: Uploading piece... Dec 31 11:56:12.264609000 (620h27m12.135692035s) [noise=true , address=45.91.116.11:4145 , piece_id=ZX5FCIOENG6476SS6RDITOCPPCDBXAYOHAOAP4Q545FEBKSSQDDA, seg_pos={0 0}, upload=543695]: Uploading piece... Dec 31 11:56:12.264666079 (620h27m12.135749114s) [noise=true , address=64.49.13.140:4505 , piece_id=YRQ2VLRASSQVB65ONBLZXVLXY7COOSYVUUNMKJO5AOAJKFFIEW2A, seg_pos={0 0}, upload=543695]: Uploading piece... Dec 31 11:56:12.265310566 (620h27m12.136393611s) [noise=true , address=145.79.227.44:4206 , piece_id=TR747XVZACWUD6WGRSROIO5OHFLVBM3XT3XTEWA74GXNXMMYMQHQ, seg_pos={0 0}, upload=543695]: Uploading piece... Dec 31 11:56:12.265745504 (620h27m12.136828549s) [noise=true , address=145.79.227.33:4894 , piece_id=TFZVFOU3GA45OWXOWVA4R2A3W47CS6DCK5K4DAMYYMIH4J5CD23A, seg_pos={0 0}, upload=543695]: Uploading piece... Dec 31 11:56:12.266321861 (620h27m12.137404896s) [noise=true , address=45.91.116.7:4013 , piece_id=A4VDKYMR74JC325EFEP22V3TOGZRMTQB4UOE5I2Y7JKQF32AIZAA, seg_pos={0 0}, upload=543695]: Uploading piece... Dec 31 11:56:12.266868933 (620h27m12.137951978s) [noise=true , address=64.49.13.169:4707 , piece_id=ZQTZNZZQFGHXU2DIVBDUMJTYRAWAPRATLOGQ6AY4A6N45276NDXQ, seg_pos={0 0}, upload=543695]: Uploading piece... Dec 31 11:56:12.267258635 (620h27m12.13834167s) [noise=true , address=77.90.31.13:4657 , piece_id=KSNYKBDG4YOLLPTKC4OFSQR6XOBOGNA64PC3BCKQLRLYGCWFCUUQ, seg_pos={0 0}, upload=543695]: Uploading piece... Dec 31 11:56:12.267587522 (620h27m12.138670567s) [noise=true , address=145.79.227.40:4058 , piece_id=F3RLQOLC5C6JRKLBZICBMGRSJENDRTTP3PGJEFAOBXNHZP4QI3IA, seg_pos={0 0}, upload=543695]: Uploading piece... Dec 31 11:56:12.268440155 (620h27m12.13952319s) [noise=true , address=145.79.227.34:4939 , piece_id=TSCIX7EBMZ72GAOPGWHMCUW4PVEONBZ56A3ODVMIMY6LZZW2XDAQ, seg_pos={0 0}, upload=543695]: Uploading piece... Dec 31 11:56:12.269297889 (620h27m12.140380924s) [noise=true , address=145.79.227.2:4134 , piece_id=HWWXLVVLSMPGUKIP5A5T5IXAB3ZCABH7OA5DIJSNDXRQU23EIFNA, seg_pos={0 0}, upload=543695]: Uploading piece... Dec 31 11:56:13.698835153 (620h27m13.569918188s) [upload=543695]: Next returned nil segment Dec 31 11:56:13.698965801 (620h27m13.570048846s) [upload=543695]: Waiting for error group managing segments... Dec 31 11:56:14.047217086 (620h27m13.918300131s) [noise=true , address=64.49.13.140:4505 , piece_id=YRQ2VLRASSQVB65ONBLZXVLXY7COOSYVUUNMKJO5AOAJKFFIEW2A, seg_pos={0 0}, upload=543695]: Done uploading piece. err: <nil> Dec 31 11:56:14.137538488 (620h27m14.008621523s) [noise=true , address=145.79.227.2:4134 , piece_id=HWWXLVVLSMPGUKIP5A5T5IXAB3ZCABH7OA5DIJSNDXRQU23EIFNA, seg_pos={0 0}, upload=543695]: Done uploading piece. err: <nil> Dec 31 11:56:14.139917508 (620h27m14.011000543s) [noise=true , address=145.79.227.46:4619 , piece_id=BV3NLVQZM457CG2ZYJYZANIDKDVYCQWC64FZGRXFFDV6FFLC2SJA, seg_pos={0 0}, upload=543695]: Done uploading piece. err: <nil> Dec 31 11:56:14.472160101 (620h27m14.343243136s) [noise=true , address=77.90.31.17:4794 , piece_id=ZCNDE7ID7HX43BDZ7KFQKZ6R5V5KUPZWCIRRUQNK65I73JMZQ2JQ, seg_pos={0 0}, upload=543695]: Done uploading piece. err: <nil> Dec 31 11:56:14.476987874 (620h27m14.348070919s) [noise=true , address=45.91.116.5:4326 , piece_id=VDLNHILWQD6ZTD7JMHQY5BDH2A4IHSJTVHJGDI4IK3OBDZSLOADA, seg_pos={0 0}, upload=543695]: Done uploading piece. err: <nil> Dec 31 11:56:14.518004714 (620h27m14.389087749s) [noise=true , address=45.91.116.7:4013 , piece_id=A4VDKYMR74JC325EFEP22V3TOGZRMTQB4UOE5I2Y7JKQF32AIZAA, seg_pos={0 0}, upload=543695]: Done uploading piece. err: <nil> Dec 31 11:56:14.540552146 (620h27m14.411635191s) [noise=true , address=77.90.31.18:4812 , piece_id=KOGGG55EKOFTTT7OOZ5AULZHU2POHF5DMMSX2RM5M65FC3Y3AXRQ, seg_pos={0 0}, upload=543695]: Done uploading piece. err: <nil> Dec 31 11:56:14.677369210 (620h27m14.548452255s) [noise=true , address=145.79.227.40:4058 , piece_id=F3RLQOLC5C6JRKLBZICBMGRSJENDRTTP3PGJEFAOBXNHZP4QI3IA, seg_pos={0 0}, upload=543695]: Done uploading piece. err: <nil> Dec 31 11:56:15.034375707 (620h27m14.905458742s) [noise=true , address=145.79.227.44:4206 , piece_id=TR747XVZACWUD6WGRSROIO5OHFLVBM3XT3XTEWA74GXNXMMYMQHQ, seg_pos={0 0}, upload=543695]: Done uploading piece. err: <nil> Dec 31 11:56:15.095619932 (620h27m14.966702967s) [noise=true , address=145.79.227.49:4358 , piece_id=RP2Y6L6CZD3XOL7XOU3HRDSYZZ6AENXT7LM6XVHZPIZAHSL26BPQ, seg_pos={0 0}, upload=543695]: Done uploading piece. err: <nil> Dec 31 11:56:15.099851950 (620h27m14.970934995s) [noise=true , address=145.79.227.43:4164 , piece_id=QQ4D2S4XAMIDBDSZNYM2G5WT4V7DR4E5WHDSYVT4SZ4YHANAY4QA, seg_pos={0 0}, upload=543695]: Done uploading piece. err: <nil> Dec 31 11:56:15.113070049 (620h27m14.984153094s) [noise=true , address=145.79.227.33:4894 , piece_id=TFZVFOU3GA45OWXOWVA4R2A3W47CS6DCK5K4DAMYYMIH4J5CD23A, seg_pos={0 0}, upload=543695]: Done uploading piece. err: <nil> Dec 31 11:56:15.127931066 (620h27m14.999014101s) [noise=true , address=145.79.227.30:4792 , piece_id=B72NRZMLRWZQRV222OBU6EPXZYYF4K6PXT6L77XZKNVZOA25NLIQ, seg_pos={0 0}, upload=543695]: Done uploading piece. err: <nil> Dec 31 11:56:15.147826669 (620h27m15.018909714s) [noise=true , address=193.149.164.143:4695 , piece_id=5CJUMEUXTJVQOCSHQKZD4ZHQBTFSMUPIQCI7QMJX2URBSUAIGIFA, seg_pos={0 0}, upload=543695]: Done uploading piece. err: <nil> Dec 31 11:56:15.147845695 (620h27m15.01892874s) [seg_pos={0 0}, upload=543695]: Segment reached optimal threshold of 14 pieces. Dec 31 11:56:15.148175553 (620h27m15.019258598s) [noise=true , address=145.79.227.34:4939 , piece_id=TSCIX7EBMZ72GAOPGWHMCUW4PVEONBZ56A3ODVMIMY6LZZW2XDAQ, seg_pos={0 0}, upload=543695]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1h7xrPMs5kHtjm7hiB4hfDaZUHFgtJNoMGqdy3HFif2Aned8eu): protocol: expected piece hash; context canceled; EOF Dec 31 11:56:15.148194439 (620h27m15.019277484s) [noise=true , address=145.79.227.34:4939 , piece_id=TSCIX7EBMZ72GAOPGWHMCUW4PVEONBZ56A3ODVMIMY6LZZW2XDAQ, seg_pos={0 0}, upload=543695]: Long tail cancellation due to optimal threshold Dec 31 11:56:15.148491545 (620h27m15.01957458s) [noise=true , address=77.90.31.13:4657 , piece_id=KSNYKBDG4YOLLPTKC4OFSQR6XOBOGNA64PC3BCKQLRLYGCWFCUUQ, seg_pos={0 0}, upload=543695]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1LLSnWuMa6W7y2jhFUqXoSPEsTyrFsKi8FQDNXF8Tv7XHbP6yt): protocol: write tcp 188.166.225.170:49614->77.90.31.13:4657: use of closed network connection; write tcp 188.166.225.170:49614->77.90.31.13:4657: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49614->77.90.31.13:4657: use of closed network connection Dec 31 11:56:15.148498648 (620h27m15.019581693s) [noise=true , address=77.90.31.13:4657 , piece_id=KSNYKBDG4YOLLPTKC4OFSQR6XOBOGNA64PC3BCKQLRLYGCWFCUUQ, seg_pos={0 0}, upload=543695]: Long tail cancellation due to optimal threshold Dec 31 11:56:15.148641420 (620h27m15.019724465s) [noise=true , address=45.91.116.11:4145 , piece_id=ZX5FCIOENG6476SS6RDITOCPPCDBXAYOHAOAP4Q545FEBKSSQDDA, seg_pos={0 0}, upload=543695]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WKV8PJxq5Kyb3ubCbrfbyYaX6fvRkPMKW2ssUteMFNHm9fP4E): protocol: expected piece hash; context canceled; EOF Dec 31 11:56:15.148647441 (620h27m15.019730476s) [noise=true , address=45.91.116.11:4145 , piece_id=ZX5FCIOENG6476SS6RDITOCPPCDBXAYOHAOAP4Q545FEBKSSQDDA, seg_pos={0 0}, upload=543695]: Long tail cancellation due to optimal threshold Dec 31 11:56:15.148750738 (620h27m15.019833773s) [noise=true , address=64.49.13.169:4707 , piece_id=ZQTZNZZQFGHXU2DIVBDUMJTYRAWAPRATLOGQ6AY4A6N45276NDXQ, seg_pos={0 0}, upload=543695]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:45002->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:45002->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45002->64.49.13.169:4707: use of closed network connection Dec 31 11:56:15.148766589 (620h27m15.019849634s) [noise=true , address=64.49.13.169:4707 , piece_id=ZQTZNZZQFGHXU2DIVBDUMJTYRAWAPRATLOGQ6AY4A6N45276NDXQ, seg_pos={0 0}, upload=543695]: Long tail cancellation due to optimal threshold Dec 31 11:56:15.148782148 (620h27m15.019865193s) [seg_pos={0 0}, upload=543695]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 11:56:15.148845759 (620h27m15.019928804s) [upload=543695]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 11:56:15.148850428 (620h27m15.019933463s) [upload=543695]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 11:56:15.442319734 (620h27m15.313402779s) [upload=543695]: Done uploading segments... Dec 31 11:56:15.442337046 (620h27m15.313420081s) [upload=543695]: Upload finished. err: <nil> Duration: 30.391695292s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................................X...X.................] duration: 24.440779361s trace: https://jaeger.dev.storj.io/trace/2e598bd65e5b1d02 FAIL (2025-12-31 12:05:18 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 24.440796123s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [.........X..X..XX....X....X.....XX.XX.X....XX.....X.....X...] duration: 5.771742735s trace: https://jaeger.dev.storj.io/trace/2aeaacdc7b2ec FAIL (2025-12-31 12:05:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 31 12:03:06.696854561 (620h34m6.567937596s) [upload=543793]: Starting upload Dec 31 12:03:06.696906339 (620h34m6.567989384s) [upload=543793]: Done starting upload Dec 31 12:03:06.697022900 (620h34m6.568105945s) [upload=543793]: Uploading segments... Dec 31 12:03:06.697101751 (620h34m6.568184786s) [upload=543793]: Got next segment. Inline: false Dec 31 12:03:06.697136497 (620h34m6.568219542s) [upload=543793]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 12:03:06.697142539 (620h34m6.568225584s) [upload=543793]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 12:03:06.941461997 (620h34m6.812545032s) [seg_pos={0 0}, upload=543793]: Begin upload segment... Dec 31 12:03:06.941527512 (620h34m6.812610557s) [seg_pos={0 0}, upload=543793]: Done begin upload segment. Dec 31 12:03:06.941595150 (620h34m6.812678185s) [noise=true , address=77.90.31.21:4878 , piece_id=6IFN4VOBLW2F3HQBWQC3VAHKU2P4EUQM232KRYBXVDSVQADA5YUA, seg_pos={0 0}, upload=543793]: Uploading piece... Dec 31 12:03:06.941627272 (620h34m6.812710317s) [noise=true , address=145.79.227.31:4798 , piece_id=WOUKZMWL6E77DEHNOFT65LE2JE77W7GFZ4U4IFTN6SEYVOQ4SBUA, seg_pos={0 0}, upload=543793]: Uploading piece... Dec 31 12:03:06.941722613 (620h34m6.812805649s) [noise=true , address=145.79.227.37:4566 , piece_id=C4LJ4JIQHTUH72K26O3YTU4CYPOO4XBH7EOMMFPIDB64JB3LZVTA, seg_pos={0 0}, upload=543793]: Uploading piece... Dec 31 12:03:06.942242333 (620h34m6.813325378s) [noise=true , address=145.79.227.8:4329 , piece_id=LXMQABMWVOOCIBKZ2EL7HTW5QTUJGB3FDTZAFELIM7DYZS42WNLQ, seg_pos={0 0}, upload=543793]: Uploading piece... Dec 31 12:03:06.942409090 (620h34m6.813492125s) [noise=true , address=45.91.116.5:4297 , piece_id=32QRF6F4NAVYHWBIOYTLKBD6RJOE3A5DO7N3THJAIPYSBADKLEBQ, seg_pos={0 0}, upload=543793]: Uploading piece... Dec 31 12:03:06.942600695 (620h34m6.81368373s) [noise=true , address=145.79.227.14:4837 , piece_id=LOEPZ4EIZKRWQYII7DUWRJHWBI7MCFHOLJLDZ5EWHPQBRAUXNCOQ, seg_pos={0 0}, upload=543793]: Uploading piece... Dec 31 12:03:06.942760429 (620h34m6.813843464s) [noise=true , address=145.79.227.45:4230 , piece_id=NTHVSF643LXDW5MQ5DQGM3BHPTIXOEL5NEKCLW6ZR7MIJLODF5LA, seg_pos={0 0}, upload=543793]: Uploading piece... Dec 31 12:03:06.943142737 (620h34m6.814225772s) [noise=true , address=145.79.227.36:4517 , piece_id=LGLPG7PMCKGSQ5TRKVQFHZB5DXNMPUKUHFV5AC7CFVACLIEIXQHA, seg_pos={0 0}, upload=543793]: Uploading piece... Dec 31 12:03:06.943205877 (620h34m6.814288912s) [noise=true , address=145.79.227.19:4078 , piece_id=K5G5LEONW3RAWGLW3JAZSK7QW6QFHROZO6PX3EF5A3AJKZAXKK6A, seg_pos={0 0}, upload=543793]: Uploading piece... Dec 31 12:03:06.943520025 (620h34m6.81460307s) [noise=true , address=145.79.227.50:4375 , piece_id=K22R6U4PNLO7ZNOBTKYVJ6URJX5CACIDPQ44DRQYQW4BADMHK4SQ, seg_pos={0 0}, upload=543793]: Uploading piece... Dec 31 12:03:06.943967728 (620h34m6.815050773s) [noise=true , address=145.79.227.13:4802 , piece_id=Y35WHD72DB5NBDPIOJVWKMZ2S7MZSKWUJZAIEKVACFML7SZBNKNQ, seg_pos={0 0}, upload=543793]: Uploading piece... Dec 31 12:03:06.944351178 (620h34m6.815434223s) [noise=true , address=77.90.31.16:4736 , piece_id=67UHO2QV3YFPWCDJJPSCHFDEUHFZU4O4YJCAK2KXYE2HMOKNQGZA, seg_pos={0 0}, upload=543793]: Uploading piece... Dec 31 12:03:06.944709330 (620h34m6.815792365s) [noise=true , address=145.79.227.22:4509 , piece_id=YC2PT7M7N7VG5FL4Q2BRDULJDY47HSKSNJ6KWYH5JZGWZTYBFOUA, seg_pos={0 0}, upload=543793]: Uploading piece... Dec 31 12:03:06.945098561 (620h34m6.816181596s) [noise=true , address=145.79.227.42:4121 , piece_id=2GLF77AV622O5PLEGOWPZR36PWHLD34W7EC24J242ZMPEMQPS7AQ, seg_pos={0 0}, upload=543793]: Uploading piece... Dec 31 12:03:06.945725214 (620h34m6.816808249s) [noise=true , address=45.91.116.6:4975 , piece_id=XMLZC7GCIQXV7YV32HKDBKBDODOVMBCORVKEV7SZC4POWLIQNMOQ, seg_pos={0 0}, upload=543793]: Uploading piece... Dec 31 12:03:06.947935113 (620h34m6.819018158s) [noise=true , address=77.90.31.11:4555 , piece_id=NN2EBC66KECKQIXTKB2ZVSM2YVLKLXCCLQWMP5RN2C5UWZM5KFFQ, seg_pos={0 0}, upload=543793]: Uploading piece... Dec 31 12:03:08.633787608 (620h34m8.504870663s) [upload=543793]: Next returned nil segment Dec 31 12:03:08.633836921 (620h34m8.504919967s) [upload=543793]: Waiting for error group managing segments... Dec 31 12:03:08.914501466 (620h34m8.785584501s) [noise=true , address=145.79.227.45:4230 , piece_id=NTHVSF643LXDW5MQ5DQGM3BHPTIXOEL5NEKCLW6ZR7MIJLODF5LA, seg_pos={0 0}, upload=543793]: Done uploading piece. err: <nil> Dec 31 12:03:08.919724370 (620h34m8.790807405s) [noise=true , address=145.79.227.22:4509 , piece_id=YC2PT7M7N7VG5FL4Q2BRDULJDY47HSKSNJ6KWYH5JZGWZTYBFOUA, seg_pos={0 0}, upload=543793]: Done uploading piece. err: <nil> Dec 31 12:03:08.927631316 (620h34m8.798714351s) [noise=true , address=145.79.227.14:4837 , piece_id=LOEPZ4EIZKRWQYII7DUWRJHWBI7MCFHOLJLDZ5EWHPQBRAUXNCOQ, seg_pos={0 0}, upload=543793]: Done uploading piece. err: <nil> Dec 31 12:03:08.932530494 (620h34m8.803613539s) [noise=true , address=145.79.227.36:4517 , piece_id=LGLPG7PMCKGSQ5TRKVQFHZB5DXNMPUKUHFV5AC7CFVACLIEIXQHA, seg_pos={0 0}, upload=543793]: Done uploading piece. err: <nil> Dec 31 12:03:08.937408452 (620h34m8.808491488s) [noise=true , address=145.79.227.37:4566 , piece_id=C4LJ4JIQHTUH72K26O3YTU4CYPOO4XBH7EOMMFPIDB64JB3LZVTA, seg_pos={0 0}, upload=543793]: Done uploading piece. err: <nil> Dec 31 12:03:08.959456975 (620h34m8.83054001s) [noise=true , address=145.79.227.31:4798 , piece_id=WOUKZMWL6E77DEHNOFT65LE2JE77W7GFZ4U4IFTN6SEYVOQ4SBUA, seg_pos={0 0}, upload=543793]: Done uploading piece. err: <nil> Dec 31 12:03:08.959912633 (620h34m8.830995688s) [noise=true , address=145.79.227.8:4329 , piece_id=LXMQABMWVOOCIBKZ2EL7HTW5QTUJGB3FDTZAFELIM7DYZS42WNLQ, seg_pos={0 0}, upload=543793]: Done uploading piece. err: <nil> Dec 31 12:03:08.960449174 (620h34m8.831532219s) [noise=true , address=145.79.227.42:4121 , piece_id=2GLF77AV622O5PLEGOWPZR36PWHLD34W7EC24J242ZMPEMQPS7AQ, seg_pos={0 0}, upload=543793]: Done uploading piece. err: <nil> Dec 31 12:03:08.961014460 (620h34m8.832097495s) [noise=true , address=145.79.227.50:4375 , piece_id=K22R6U4PNLO7ZNOBTKYVJ6URJX5CACIDPQ44DRQYQW4BADMHK4SQ, seg_pos={0 0}, upload=543793]: Done uploading piece. err: <nil> Dec 31 12:03:08.999315790 (620h34m8.870398825s) [noise=true , address=145.79.227.19:4078 , piece_id=K5G5LEONW3RAWGLW3JAZSK7QW6QFHROZO6PX3EF5A3AJKZAXKK6A, seg_pos={0 0}, upload=543793]: Done uploading piece. err: <nil> Dec 31 12:03:09.017297168 (620h34m8.888380204s) [noise=true , address=145.79.227.13:4802 , piece_id=Y35WHD72DB5NBDPIOJVWKMZ2S7MZSKWUJZAIEKVACFML7SZBNKNQ, seg_pos={0 0}, upload=543793]: Done uploading piece. err: <nil> Dec 31 12:03:09.181590309 (620h34m9.052673344s) [noise=true , address=77.90.31.21:4878 , piece_id=6IFN4VOBLW2F3HQBWQC3VAHKU2P4EUQM232KRYBXVDSVQADA5YUA, seg_pos={0 0}, upload=543793]: Done uploading piece. err: <nil> Dec 31 12:03:09.243812086 (620h34m9.114895121s) [noise=true , address=77.90.31.11:4555 , piece_id=NN2EBC66KECKQIXTKB2ZVSM2YVLKLXCCLQWMP5RN2C5UWZM5KFFQ, seg_pos={0 0}, upload=543793]: Done uploading piece. err: <nil> Dec 31 12:03:09.243837224 (620h34m9.114920269s) [seg_pos={0 0}, upload=543793]: Segment reached optimal threshold of 13 pieces. Dec 31 12:03:09.244070898 (620h34m9.115153943s) [noise=true , address=45.91.116.5:4297 , piece_id=32QRF6F4NAVYHWBIOYTLKBD6RJOE3A5DO7N3THJAIPYSBADKLEBQ, seg_pos={0 0}, upload=543793]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 31 12:03:09.244102789 (620h34m9.115185824s) [noise=true , address=45.91.116.5:4297 , piece_id=32QRF6F4NAVYHWBIOYTLKBD6RJOE3A5DO7N3THJAIPYSBADKLEBQ, seg_pos={0 0}, upload=543793]: Long tail cancellation due to optimal threshold Dec 31 12:03:09.244379997 (620h34m9.115463032s) [noise=true , address=77.90.31.16:4736 , piece_id=67UHO2QV3YFPWCDJJPSCHFDEUHFZU4O4YJCAK2KXYE2HMOKNQGZA, seg_pos={0 0}, upload=543793]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 31 12:03:09.244387681 (620h34m9.115470727s) [noise=true , address=77.90.31.16:4736 , piece_id=67UHO2QV3YFPWCDJJPSCHFDEUHFZU4O4YJCAK2KXYE2HMOKNQGZA, seg_pos={0 0}, upload=543793]: Long tail cancellation due to optimal threshold Dec 31 12:03:09.244536826 (620h34m9.115619861s) [noise=true , address=45.91.116.6:4975 , piece_id=XMLZC7GCIQXV7YV32HKDBKBDODOVMBCORVKEV7SZC4POWLIQNMOQ, seg_pos={0 0}, upload=543793]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 31 12:03:09.244545593 (620h34m9.115628638s) [noise=true , address=45.91.116.6:4975 , piece_id=XMLZC7GCIQXV7YV32HKDBKBDODOVMBCORVKEV7SZC4POWLIQNMOQ, seg_pos={0 0}, upload=543793]: Long tail cancellation due to optimal threshold Dec 31 12:03:09.244565681 (620h34m9.115648716s) [seg_pos={0 0}, upload=543793]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 12:03:09.244644411 (620h34m9.115727456s) [upload=543793]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 12:03:09.244650142 (620h34m9.115733187s) [upload=543793]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 12:03:09.534801749 (620h34m9.405884784s) [upload=543793]: Done uploading segments... Dec 31 12:03:09.534822359 (620h34m9.405905404s) [upload=543793]: Upload finished. err: <nil> Duration: 5.771814331s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................................X...X................] duration: 19.963700848s trace: https://jaeger.dev.storj.io/trace/2e598bd65e5b1d02 FAIL (2025-12-31 12:04:54 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 19.963727118s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [.X....X................X..........X.X......X.............X..] duration: 3.466562289s trace: https://jaeger.dev.storj.io/trace/2c96062d1b5b5a44 FAIL (2025-12-31 12:04:48 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.466596323s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [X.........X..X..XX....X....X.....XX.XX.X....XX.....X.....X..] duration: 5.873281726s trace: https://jaeger.dev.storj.io/trace/2aeaacdc7b2ec FAIL (2025-12-31 12:04:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 31 12:03:06.696854561 (620h34m6.567937596s) [upload=543793]: Starting upload Dec 31 12:03:06.696906339 (620h34m6.567989384s) [upload=543793]: Done starting upload Dec 31 12:03:06.697022900 (620h34m6.568105945s) [upload=543793]: Uploading segments... Dec 31 12:03:06.697101751 (620h34m6.568184786s) [upload=543793]: Got next segment. Inline: false Dec 31 12:03:06.697136497 (620h34m6.568219542s) [upload=543793]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 12:03:06.697142539 (620h34m6.568225584s) [upload=543793]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 12:03:06.941461997 (620h34m6.812545032s) [seg_pos={0 0}, upload=543793]: Begin upload segment... Dec 31 12:03:06.941527512 (620h34m6.812610557s) [seg_pos={0 0}, upload=543793]: Done begin upload segment. Dec 31 12:03:06.941595150 (620h34m6.812678185s) [noise=true , address=77.90.31.21:4878 , piece_id=6IFN4VOBLW2F3HQBWQC3VAHKU2P4EUQM232KRYBXVDSVQADA5YUA, seg_pos={0 0}, upload=543793]: Uploading piece... Dec 31 12:03:06.941627272 (620h34m6.812710317s) [noise=true , address=145.79.227.31:4798 , piece_id=WOUKZMWL6E77DEHNOFT65LE2JE77W7GFZ4U4IFTN6SEYVOQ4SBUA, seg_pos={0 0}, upload=543793]: Uploading piece... Dec 31 12:03:06.941722613 (620h34m6.812805649s) [noise=true , address=145.79.227.37:4566 , piece_id=C4LJ4JIQHTUH72K26O3YTU4CYPOO4XBH7EOMMFPIDB64JB3LZVTA, seg_pos={0 0}, upload=543793]: Uploading piece... Dec 31 12:03:06.942242333 (620h34m6.813325378s) [noise=true , address=145.79.227.8:4329 , piece_id=LXMQABMWVOOCIBKZ2EL7HTW5QTUJGB3FDTZAFELIM7DYZS42WNLQ, seg_pos={0 0}, upload=543793]: Uploading piece... Dec 31 12:03:06.942409090 (620h34m6.813492125s) [noise=true , address=45.91.116.5:4297 , piece_id=32QRF6F4NAVYHWBIOYTLKBD6RJOE3A5DO7N3THJAIPYSBADKLEBQ, seg_pos={0 0}, upload=543793]: Uploading piece... Dec 31 12:03:06.942600695 (620h34m6.81368373s) [noise=true , address=145.79.227.14:4837 , piece_id=LOEPZ4EIZKRWQYII7DUWRJHWBI7MCFHOLJLDZ5EWHPQBRAUXNCOQ, seg_pos={0 0}, upload=543793]: Uploading piece... Dec 31 12:03:06.942760429 (620h34m6.813843464s) [noise=true , address=145.79.227.45:4230 , piece_id=NTHVSF643LXDW5MQ5DQGM3BHPTIXOEL5NEKCLW6ZR7MIJLODF5LA, seg_pos={0 0}, upload=543793]: Uploading piece... Dec 31 12:03:06.943142737 (620h34m6.814225772s) [noise=true , address=145.79.227.36:4517 , piece_id=LGLPG7PMCKGSQ5TRKVQFHZB5DXNMPUKUHFV5AC7CFVACLIEIXQHA, seg_pos={0 0}, upload=543793]: Uploading piece... Dec 31 12:03:06.943205877 (620h34m6.814288912s) [noise=true , address=145.79.227.19:4078 , piece_id=K5G5LEONW3RAWGLW3JAZSK7QW6QFHROZO6PX3EF5A3AJKZAXKK6A, seg_pos={0 0}, upload=543793]: Uploading piece... Dec 31 12:03:06.943520025 (620h34m6.81460307s) [noise=true , address=145.79.227.50:4375 , piece_id=K22R6U4PNLO7ZNOBTKYVJ6URJX5CACIDPQ44DRQYQW4BADMHK4SQ, seg_pos={0 0}, upload=543793]: Uploading piece... Dec 31 12:03:06.943967728 (620h34m6.815050773s) [noise=true , address=145.79.227.13:4802 , piece_id=Y35WHD72DB5NBDPIOJVWKMZ2S7MZSKWUJZAIEKVACFML7SZBNKNQ, seg_pos={0 0}, upload=543793]: Uploading piece... Dec 31 12:03:06.944351178 (620h34m6.815434223s) [noise=true , address=77.90.31.16:4736 , piece_id=67UHO2QV3YFPWCDJJPSCHFDEUHFZU4O4YJCAK2KXYE2HMOKNQGZA, seg_pos={0 0}, upload=543793]: Uploading piece... Dec 31 12:03:06.944709330 (620h34m6.815792365s) [noise=true , address=145.79.227.22:4509 , piece_id=YC2PT7M7N7VG5FL4Q2BRDULJDY47HSKSNJ6KWYH5JZGWZTYBFOUA, seg_pos={0 0}, upload=543793]: Uploading piece... Dec 31 12:03:06.945098561 (620h34m6.816181596s) [noise=true , address=145.79.227.42:4121 , piece_id=2GLF77AV622O5PLEGOWPZR36PWHLD34W7EC24J242ZMPEMQPS7AQ, seg_pos={0 0}, upload=543793]: Uploading piece... Dec 31 12:03:06.945725214 (620h34m6.816808249s) [noise=true , address=45.91.116.6:4975 , piece_id=XMLZC7GCIQXV7YV32HKDBKBDODOVMBCORVKEV7SZC4POWLIQNMOQ, seg_pos={0 0}, upload=543793]: Uploading piece... Dec 31 12:03:06.947935113 (620h34m6.819018158s) [noise=true , address=77.90.31.11:4555 , piece_id=NN2EBC66KECKQIXTKB2ZVSM2YVLKLXCCLQWMP5RN2C5UWZM5KFFQ, seg_pos={0 0}, upload=543793]: Uploading piece... Dec 31 12:03:08.633787608 (620h34m8.504870663s) [upload=543793]: Next returned nil segment Dec 31 12:03:08.633836921 (620h34m8.504919967s) [upload=543793]: Waiting for error group managing segments... Dec 31 12:03:08.914501466 (620h34m8.785584501s) [noise=true , address=145.79.227.45:4230 , piece_id=NTHVSF643LXDW5MQ5DQGM3BHPTIXOEL5NEKCLW6ZR7MIJLODF5LA, seg_pos={0 0}, upload=543793]: Done uploading piece. err: <nil> Dec 31 12:03:08.919724370 (620h34m8.790807405s) [noise=true , address=145.79.227.22:4509 , piece_id=YC2PT7M7N7VG5FL4Q2BRDULJDY47HSKSNJ6KWYH5JZGWZTYBFOUA, seg_pos={0 0}, upload=543793]: Done uploading piece. err: <nil> Dec 31 12:03:08.927631316 (620h34m8.798714351s) [noise=true , address=145.79.227.14:4837 , piece_id=LOEPZ4EIZKRWQYII7DUWRJHWBI7MCFHOLJLDZ5EWHPQBRAUXNCOQ, seg_pos={0 0}, upload=543793]: Done uploading piece. err: <nil> Dec 31 12:03:08.932530494 (620h34m8.803613539s) [noise=true , address=145.79.227.36:4517 , piece_id=LGLPG7PMCKGSQ5TRKVQFHZB5DXNMPUKUHFV5AC7CFVACLIEIXQHA, seg_pos={0 0}, upload=543793]: Done uploading piece. err: <nil> Dec 31 12:03:08.937408452 (620h34m8.808491488s) [noise=true , address=145.79.227.37:4566 , piece_id=C4LJ4JIQHTUH72K26O3YTU4CYPOO4XBH7EOMMFPIDB64JB3LZVTA, seg_pos={0 0}, upload=543793]: Done uploading piece. err: <nil> Dec 31 12:03:08.959456975 (620h34m8.83054001s) [noise=true , address=145.79.227.31:4798 , piece_id=WOUKZMWL6E77DEHNOFT65LE2JE77W7GFZ4U4IFTN6SEYVOQ4SBUA, seg_pos={0 0}, upload=543793]: Done uploading piece. err: <nil> Dec 31 12:03:08.959912633 (620h34m8.830995688s) [noise=true , address=145.79.227.8:4329 , piece_id=LXMQABMWVOOCIBKZ2EL7HTW5QTUJGB3FDTZAFELIM7DYZS42WNLQ, seg_pos={0 0}, upload=543793]: Done uploading piece. err: <nil> Dec 31 12:03:08.960449174 (620h34m8.831532219s) [noise=true , address=145.79.227.42:4121 , piece_id=2GLF77AV622O5PLEGOWPZR36PWHLD34W7EC24J242ZMPEMQPS7AQ, seg_pos={0 0}, upload=543793]: Done uploading piece. err: <nil> Dec 31 12:03:08.961014460 (620h34m8.832097495s) [noise=true , address=145.79.227.50:4375 , piece_id=K22R6U4PNLO7ZNOBTKYVJ6URJX5CACIDPQ44DRQYQW4BADMHK4SQ, seg_pos={0 0}, upload=543793]: Done uploading piece. err: <nil> Dec 31 12:03:08.999315790 (620h34m8.870398825s) [noise=true , address=145.79.227.19:4078 , piece_id=K5G5LEONW3RAWGLW3JAZSK7QW6QFHROZO6PX3EF5A3AJKZAXKK6A, seg_pos={0 0}, upload=543793]: Done uploading piece. err: <nil> Dec 31 12:03:09.017297168 (620h34m8.888380204s) [noise=true , address=145.79.227.13:4802 , piece_id=Y35WHD72DB5NBDPIOJVWKMZ2S7MZSKWUJZAIEKVACFML7SZBNKNQ, seg_pos={0 0}, upload=543793]: Done uploading piece. err: <nil> Dec 31 12:03:09.181590309 (620h34m9.052673344s) [noise=true , address=77.90.31.21:4878 , piece_id=6IFN4VOBLW2F3HQBWQC3VAHKU2P4EUQM232KRYBXVDSVQADA5YUA, seg_pos={0 0}, upload=543793]: Done uploading piece. err: <nil> Dec 31 12:03:09.243812086 (620h34m9.114895121s) [noise=true , address=77.90.31.11:4555 , piece_id=NN2EBC66KECKQIXTKB2ZVSM2YVLKLXCCLQWMP5RN2C5UWZM5KFFQ, seg_pos={0 0}, upload=543793]: Done uploading piece. err: <nil> Dec 31 12:03:09.243837224 (620h34m9.114920269s) [seg_pos={0 0}, upload=543793]: Segment reached optimal threshold of 13 pieces. Dec 31 12:03:09.244070898 (620h34m9.115153943s) [noise=true , address=45.91.116.5:4297 , piece_id=32QRF6F4NAVYHWBIOYTLKBD6RJOE3A5DO7N3THJAIPYSBADKLEBQ, seg_pos={0 0}, upload=543793]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 31 12:03:09.244102789 (620h34m9.115185824s) [noise=true , address=45.91.116.5:4297 , piece_id=32QRF6F4NAVYHWBIOYTLKBD6RJOE3A5DO7N3THJAIPYSBADKLEBQ, seg_pos={0 0}, upload=543793]: Long tail cancellation due to optimal threshold Dec 31 12:03:09.244379997 (620h34m9.115463032s) [noise=true , address=77.90.31.16:4736 , piece_id=67UHO2QV3YFPWCDJJPSCHFDEUHFZU4O4YJCAK2KXYE2HMOKNQGZA, seg_pos={0 0}, upload=543793]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 31 12:03:09.244387681 (620h34m9.115470727s) [noise=true , address=77.90.31.16:4736 , piece_id=67UHO2QV3YFPWCDJJPSCHFDEUHFZU4O4YJCAK2KXYE2HMOKNQGZA, seg_pos={0 0}, upload=543793]: Long tail cancellation due to optimal threshold Dec 31 12:03:09.244536826 (620h34m9.115619861s) [noise=true , address=45.91.116.6:4975 , piece_id=XMLZC7GCIQXV7YV32HKDBKBDODOVMBCORVKEV7SZC4POWLIQNMOQ, seg_pos={0 0}, upload=543793]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 31 12:03:09.244545593 (620h34m9.115628638s) [noise=true , address=45.91.116.6:4975 , piece_id=XMLZC7GCIQXV7YV32HKDBKBDODOVMBCORVKEV7SZC4POWLIQNMOQ, seg_pos={0 0}, upload=543793]: Long tail cancellation due to optimal threshold Dec 31 12:03:09.244565681 (620h34m9.115648716s) [seg_pos={0 0}, upload=543793]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 12:03:09.244644411 (620h34m9.115727456s) [upload=543793]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 12:03:09.244650142 (620h34m9.115733187s) [upload=543793]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 12:03:09.534801749 (620h34m9.405884784s) [upload=543793]: Done uploading segments... Dec 31 12:03:09.534822359 (620h34m9.405905404s) [upload=543793]: Upload finished. err: <nil> Duration: 5.873304399s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................................X...X...............] duration: 23.88090421s trace: https://jaeger.dev.storj.io/trace/2e598bd65e5b1d02 FAIL (2025-12-31 12:04:26 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 23.880932654s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [....X..............................................X........] duration: 33.391356657s trace: https://jaeger.dev.storj.io/trace/3696061a2156809c FAIL (2025-12-31 12:04:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== .17:4794 , piece_id=ZCNDE7ID7HX43BDZ7KFQKZ6R5V5KUPZWCIRRUQNK65I73JMZQ2JQ, seg_pos={0 0}, upload=543695]: Uploading piece... Dec 31 11:56:12.262721887 (620h27m12.133804932s) [noise=true , address=193.149.164.143:4695 , piece_id=5CJUMEUXTJVQOCSHQKZD4ZHQBTFSMUPIQCI7QMJX2URBSUAIGIFA, seg_pos={0 0}, upload=543695]: Uploading piece... Dec 31 11:56:12.263130455 (620h27m12.1342135s) [noise=true , address=45.91.116.5:4326 , piece_id=VDLNHILWQD6ZTD7JMHQY5BDH2A4IHSJTVHJGDI4IK3OBDZSLOADA, seg_pos={0 0}, upload=543695]: Uploading piece... Dec 31 11:56:12.264175605 (620h27m12.13525865s) [noise=true , address=77.90.31.18:4812 , piece_id=KOGGG55EKOFTTT7OOZ5AULZHU2POHF5DMMSX2RM5M65FC3Y3AXRQ, seg_pos={0 0}, upload=543695]: Uploading piece... Dec 31 11:56:12.264193529 (620h27m12.135276574s) [noise=true , address=145.79.227.49:4358 , piece_id=RP2Y6L6CZD3XOL7XOU3HRDSYZZ6AENXT7LM6XVHZPIZAHSL26BPQ, seg_pos={0 0}, upload=543695]: Uploading piece... Dec 31 11:56:12.264609000 (620h27m12.135692035s) [noise=true , address=45.91.116.11:4145 , piece_id=ZX5FCIOENG6476SS6RDITOCPPCDBXAYOHAOAP4Q545FEBKSSQDDA, seg_pos={0 0}, upload=543695]: Uploading piece... Dec 31 11:56:12.264666079 (620h27m12.135749114s) [noise=true , address=64.49.13.140:4505 , piece_id=YRQ2VLRASSQVB65ONBLZXVLXY7COOSYVUUNMKJO5AOAJKFFIEW2A, seg_pos={0 0}, upload=543695]: Uploading piece... Dec 31 11:56:12.265310566 (620h27m12.136393611s) [noise=true , address=145.79.227.44:4206 , piece_id=TR747XVZACWUD6WGRSROIO5OHFLVBM3XT3XTEWA74GXNXMMYMQHQ, seg_pos={0 0}, upload=543695]: Uploading piece... Dec 31 11:56:12.265745504 (620h27m12.136828549s) [noise=true , address=145.79.227.33:4894 , piece_id=TFZVFOU3GA45OWXOWVA4R2A3W47CS6DCK5K4DAMYYMIH4J5CD23A, seg_pos={0 0}, upload=543695]: Uploading piece... Dec 31 11:56:12.266321861 (620h27m12.137404896s) [noise=true , address=45.91.116.7:4013 , piece_id=A4VDKYMR74JC325EFEP22V3TOGZRMTQB4UOE5I2Y7JKQF32AIZAA, seg_pos={0 0}, upload=543695]: Uploading piece... Dec 31 11:56:12.266868933 (620h27m12.137951978s) [noise=true , address=64.49.13.169:4707 , piece_id=ZQTZNZZQFGHXU2DIVBDUMJTYRAWAPRATLOGQ6AY4A6N45276NDXQ, seg_pos={0 0}, upload=543695]: Uploading piece... Dec 31 11:56:12.267258635 (620h27m12.13834167s) [noise=true , address=77.90.31.13:4657 , piece_id=KSNYKBDG4YOLLPTKC4OFSQR6XOBOGNA64PC3BCKQLRLYGCWFCUUQ, seg_pos={0 0}, upload=543695]: Uploading piece... Dec 31 11:56:12.267587522 (620h27m12.138670567s) [noise=true , address=145.79.227.40:4058 , piece_id=F3RLQOLC5C6JRKLBZICBMGRSJENDRTTP3PGJEFAOBXNHZP4QI3IA, seg_pos={0 0}, upload=543695]: Uploading piece... Dec 31 11:56:12.268440155 (620h27m12.13952319s) [noise=true , address=145.79.227.34:4939 , piece_id=TSCIX7EBMZ72GAOPGWHMCUW4PVEONBZ56A3ODVMIMY6LZZW2XDAQ, seg_pos={0 0}, upload=543695]: Uploading piece... Dec 31 11:56:12.269297889 (620h27m12.140380924s) [noise=true , address=145.79.227.2:4134 , piece_id=HWWXLVVLSMPGUKIP5A5T5IXAB3ZCABH7OA5DIJSNDXRQU23EIFNA, seg_pos={0 0}, upload=543695]: Uploading piece... Dec 31 11:56:13.698835153 (620h27m13.569918188s) [upload=543695]: Next returned nil segment Dec 31 11:56:13.698965801 (620h27m13.570048846s) [upload=543695]: Waiting for error group managing segments... Dec 31 11:56:14.047217086 (620h27m13.918300131s) [noise=true , address=64.49.13.140:4505 , piece_id=YRQ2VLRASSQVB65ONBLZXVLXY7COOSYVUUNMKJO5AOAJKFFIEW2A, seg_pos={0 0}, upload=543695]: Done uploading piece. err: <nil> Dec 31 11:56:14.137538488 (620h27m14.008621523s) [noise=true , address=145.79.227.2:4134 , piece_id=HWWXLVVLSMPGUKIP5A5T5IXAB3ZCABH7OA5DIJSNDXRQU23EIFNA, seg_pos={0 0}, upload=543695]: Done uploading piece. err: <nil> Dec 31 11:56:14.139917508 (620h27m14.011000543s) [noise=true , address=145.79.227.46:4619 , piece_id=BV3NLVQZM457CG2ZYJYZANIDKDVYCQWC64FZGRXFFDV6FFLC2SJA, seg_pos={0 0}, upload=543695]: Done uploading piece. err: <nil> Dec 31 11:56:14.472160101 (620h27m14.343243136s) [noise=true , address=77.90.31.17:4794 , piece_id=ZCNDE7ID7HX43BDZ7KFQKZ6R5V5KUPZWCIRRUQNK65I73JMZQ2JQ, seg_pos={0 0}, upload=543695]: Done uploading piece. err: <nil> Dec 31 11:56:14.476987874 (620h27m14.348070919s) [noise=true , address=45.91.116.5:4326 , piece_id=VDLNHILWQD6ZTD7JMHQY5BDH2A4IHSJTVHJGDI4IK3OBDZSLOADA, seg_pos={0 0}, upload=543695]: Done uploading piece. err: <nil> Dec 31 11:56:14.518004714 (620h27m14.389087749s) [noise=true , address=45.91.116.7:4013 , piece_id=A4VDKYMR74JC325EFEP22V3TOGZRMTQB4UOE5I2Y7JKQF32AIZAA, seg_pos={0 0}, upload=543695]: Done uploading piece. err: <nil> Dec 31 11:56:14.540552146 (620h27m14.411635191s) [noise=true , address=77.90.31.18:4812 , piece_id=KOGGG55EKOFTTT7OOZ5AULZHU2POHF5DMMSX2RM5M65FC3Y3AXRQ, seg_pos={0 0}, upload=543695]: Done uploading piece. err: <nil> Dec 31 11:56:14.677369210 (620h27m14.548452255s) [noise=true , address=145.79.227.40:4058 , piece_id=F3RLQOLC5C6JRKLBZICBMGRSJENDRTTP3PGJEFAOBXNHZP4QI3IA, seg_pos={0 0}, upload=543695]: Done uploading piece. err: <nil> Dec 31 11:56:15.034375707 (620h27m14.905458742s) [noise=true , address=145.79.227.44:4206 , piece_id=TR747XVZACWUD6WGRSROIO5OHFLVBM3XT3XTEWA74GXNXMMYMQHQ, seg_pos={0 0}, upload=543695]: Done uploading piece. err: <nil> Dec 31 11:56:15.095619932 (620h27m14.966702967s) [noise=true , address=145.79.227.49:4358 , piece_id=RP2Y6L6CZD3XOL7XOU3HRDSYZZ6AENXT7LM6XVHZPIZAHSL26BPQ, seg_pos={0 0}, upload=543695]: Done uploading piece. err: <nil> Dec 31 11:56:15.099851950 (620h27m14.970934995s) [noise=true , address=145.79.227.43:4164 , piece_id=QQ4D2S4XAMIDBDSZNYM2G5WT4V7DR4E5WHDSYVT4SZ4YHANAY4QA, seg_pos={0 0}, upload=543695]: Done uploading piece. err: <nil> Dec 31 11:56:15.113070049 (620h27m14.984153094s) [noise=true , address=145.79.227.33:4894 , piece_id=TFZVFOU3GA45OWXOWVA4R2A3W47CS6DCK5K4DAMYYMIH4J5CD23A, seg_pos={0 0}, upload=543695]: Done uploading piece. err: <nil> Dec 31 11:56:15.127931066 (620h27m14.999014101s) [noise=true , address=145.79.227.30:4792 , piece_id=B72NRZMLRWZQRV222OBU6EPXZYYF4K6PXT6L77XZKNVZOA25NLIQ, seg_pos={0 0}, upload=543695]: Done uploading piece. err: <nil> Dec 31 11:56:15.147826669 (620h27m15.018909714s) [noise=true , address=193.149.164.143:4695 , piece_id=5CJUMEUXTJVQOCSHQKZD4ZHQBTFSMUPIQCI7QMJX2URBSUAIGIFA, seg_pos={0 0}, upload=543695]: Done uploading piece. err: <nil> Dec 31 11:56:15.147845695 (620h27m15.01892874s) [seg_pos={0 0}, upload=543695]: Segment reached optimal threshold of 14 pieces. Dec 31 11:56:15.148175553 (620h27m15.019258598s) [noise=true , address=145.79.227.34:4939 , piece_id=TSCIX7EBMZ72GAOPGWHMCUW4PVEONBZ56A3ODVMIMY6LZZW2XDAQ, seg_pos={0 0}, upload=543695]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1h7xrPMs5kHtjm7hiB4hfDaZUHFgtJNoMGqdy3HFif2Aned8eu): protocol: expected piece hash; context canceled; EOF Dec 31 11:56:15.148194439 (620h27m15.019277484s) [noise=true , address=145.79.227.34:4939 , piece_id=TSCIX7EBMZ72GAOPGWHMCUW4PVEONBZ56A3ODVMIMY6LZZW2XDAQ, seg_pos={0 0}, upload=543695]: Long tail cancellation due to optimal threshold Dec 31 11:56:15.148491545 (620h27m15.01957458s) [noise=true , address=77.90.31.13:4657 , piece_id=KSNYKBDG4YOLLPTKC4OFSQR6XOBOGNA64PC3BCKQLRLYGCWFCUUQ, seg_pos={0 0}, upload=543695]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1LLSnWuMa6W7y2jhFUqXoSPEsTyrFsKi8FQDNXF8Tv7XHbP6yt): protocol: write tcp 188.166.225.170:49614->77.90.31.13:4657: use of closed network connection; write tcp 188.166.225.170:49614->77.90.31.13:4657: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49614->77.90.31.13:4657: use of closed network connection Dec 31 11:56:15.148498648 (620h27m15.019581693s) [noise=true , address=77.90.31.13:4657 , piece_id=KSNYKBDG4YOLLPTKC4OFSQR6XOBOGNA64PC3BCKQLRLYGCWFCUUQ, seg_pos={0 0}, upload=543695]: Long tail cancellation due to optimal threshold Dec 31 11:56:15.148641420 (620h27m15.019724465s) [noise=true , address=45.91.116.11:4145 , piece_id=ZX5FCIOENG6476SS6RDITOCPPCDBXAYOHAOAP4Q545FEBKSSQDDA, seg_pos={0 0}, upload=543695]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WKV8PJxq5Kyb3ubCbrfbyYaX6fvRkPMKW2ssUteMFNHm9fP4E): protocol: expected piece hash; context canceled; EOF Dec 31 11:56:15.148647441 (620h27m15.019730476s) [noise=true , address=45.91.116.11:4145 , piece_id=ZX5FCIOENG6476SS6RDITOCPPCDBXAYOHAOAP4Q545FEBKSSQDDA, seg_pos={0 0}, upload=543695]: Long tail cancellation due to optimal threshold Dec 31 11:56:15.148750738 (620h27m15.019833773s) [noise=true , address=64.49.13.169:4707 , piece_id=ZQTZNZZQFGHXU2DIVBDUMJTYRAWAPRATLOGQ6AY4A6N45276NDXQ, seg_pos={0 0}, upload=543695]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:45002->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:45002->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45002->64.49.13.169:4707: use of closed network connection Dec 31 11:56:15.148766589 (620h27m15.019849634s) [noise=true , address=64.49.13.169:4707 , piece_id=ZQTZNZZQFGHXU2DIVBDUMJTYRAWAPRATLOGQ6AY4A6N45276NDXQ, seg_pos={0 0}, upload=543695]: Long tail cancellation due to optimal threshold Dec 31 11:56:15.148782148 (620h27m15.019865193s) [seg_pos={0 0}, upload=543695]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 11:56:15.148845759 (620h27m15.019928804s) [upload=543695]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 11:56:15.148850428 (620h27m15.019933463s) [upload=543695]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 11:56:15.442319734 (620h27m15.313402779s) [upload=543695]: Done uploading segments... Dec 31 11:56:15.442337046 (620h27m15.313420081s) [upload=543695]: Upload finished. err: <nil> Duration: 33.391484571s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................................X...X..............] duration: 23.606563924s trace: https://jaeger.dev.storj.io/trace/2e598bd65e5b1d02 FAIL (2025-12-31 12:03:57 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 23.606591366s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [.X.........X..X..XX....X....X.....XX.XX.X....XX.....X.....X.] duration: 13.324291084s trace: https://jaeger.dev.storj.io/trace/2aeaacdc7b2ec FAIL (2025-12-31 12:03:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 31 12:03:06.696854561 (620h34m6.567937596s) [upload=543793]: Starting upload Dec 31 12:03:06.696906339 (620h34m6.567989384s) [upload=543793]: Done starting upload Dec 31 12:03:06.697022900 (620h34m6.568105945s) [upload=543793]: Uploading segments... Dec 31 12:03:06.697101751 (620h34m6.568184786s) [upload=543793]: Got next segment. Inline: false Dec 31 12:03:06.697136497 (620h34m6.568219542s) [upload=543793]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 12:03:06.697142539 (620h34m6.568225584s) [upload=543793]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 12:03:06.941461997 (620h34m6.812545032s) [seg_pos={0 0}, upload=543793]: Begin upload segment... Dec 31 12:03:06.941527512 (620h34m6.812610557s) [seg_pos={0 0}, upload=543793]: Done begin upload segment. Dec 31 12:03:06.941595150 (620h34m6.812678185s) [noise=true , address=77.90.31.21:4878 , piece_id=6IFN4VOBLW2F3HQBWQC3VAHKU2P4EUQM232KRYBXVDSVQADA5YUA, seg_pos={0 0}, upload=543793]: Uploading piece... Dec 31 12:03:06.941627272 (620h34m6.812710317s) [noise=true , address=145.79.227.31:4798 , piece_id=WOUKZMWL6E77DEHNOFT65LE2JE77W7GFZ4U4IFTN6SEYVOQ4SBUA, seg_pos={0 0}, upload=543793]: Uploading piece... Dec 31 12:03:06.941722613 (620h34m6.812805649s) [noise=true , address=145.79.227.37:4566 , piece_id=C4LJ4JIQHTUH72K26O3YTU4CYPOO4XBH7EOMMFPIDB64JB3LZVTA, seg_pos={0 0}, upload=543793]: Uploading piece... Dec 31 12:03:06.942242333 (620h34m6.813325378s) [noise=true , address=145.79.227.8:4329 , piece_id=LXMQABMWVOOCIBKZ2EL7HTW5QTUJGB3FDTZAFELIM7DYZS42WNLQ, seg_pos={0 0}, upload=543793]: Uploading piece... Dec 31 12:03:06.942409090 (620h34m6.813492125s) [noise=true , address=45.91.116.5:4297 , piece_id=32QRF6F4NAVYHWBIOYTLKBD6RJOE3A5DO7N3THJAIPYSBADKLEBQ, seg_pos={0 0}, upload=543793]: Uploading piece... Dec 31 12:03:06.942600695 (620h34m6.81368373s) [noise=true , address=145.79.227.14:4837 , piece_id=LOEPZ4EIZKRWQYII7DUWRJHWBI7MCFHOLJLDZ5EWHPQBRAUXNCOQ, seg_pos={0 0}, upload=543793]: Uploading piece... Dec 31 12:03:06.942760429 (620h34m6.813843464s) [noise=true , address=145.79.227.45:4230 , piece_id=NTHVSF643LXDW5MQ5DQGM3BHPTIXOEL5NEKCLW6ZR7MIJLODF5LA, seg_pos={0 0}, upload=543793]: Uploading piece... Dec 31 12:03:06.943142737 (620h34m6.814225772s) [noise=true , address=145.79.227.36:4517 , piece_id=LGLPG7PMCKGSQ5TRKVQFHZB5DXNMPUKUHFV5AC7CFVACLIEIXQHA, seg_pos={0 0}, upload=543793]: Uploading piece... Dec 31 12:03:06.943205877 (620h34m6.814288912s) [noise=true , address=145.79.227.19:4078 , piece_id=K5G5LEONW3RAWGLW3JAZSK7QW6QFHROZO6PX3EF5A3AJKZAXKK6A, seg_pos={0 0}, upload=543793]: Uploading piece... Dec 31 12:03:06.943520025 (620h34m6.81460307s) [noise=true , address=145.79.227.50:4375 , piece_id=K22R6U4PNLO7ZNOBTKYVJ6URJX5CACIDPQ44DRQYQW4BADMHK4SQ, seg_pos={0 0}, upload=543793]: Uploading piece... Dec 31 12:03:06.943967728 (620h34m6.815050773s) [noise=true , address=145.79.227.13:4802 , piece_id=Y35WHD72DB5NBDPIOJVWKMZ2S7MZSKWUJZAIEKVACFML7SZBNKNQ, seg_pos={0 0}, upload=543793]: Uploading piece... Dec 31 12:03:06.944351178 (620h34m6.815434223s) [noise=true , address=77.90.31.16:4736 , piece_id=67UHO2QV3YFPWCDJJPSCHFDEUHFZU4O4YJCAK2KXYE2HMOKNQGZA, seg_pos={0 0}, upload=543793]: Uploading piece... Dec 31 12:03:06.944709330 (620h34m6.815792365s) [noise=true , address=145.79.227.22:4509 , piece_id=YC2PT7M7N7VG5FL4Q2BRDULJDY47HSKSNJ6KWYH5JZGWZTYBFOUA, seg_pos={0 0}, upload=543793]: Uploading piece... Dec 31 12:03:06.945098561 (620h34m6.816181596s) [noise=true , address=145.79.227.42:4121 , piece_id=2GLF77AV622O5PLEGOWPZR36PWHLD34W7EC24J242ZMPEMQPS7AQ, seg_pos={0 0}, upload=543793]: Uploading piece... Dec 31 12:03:06.945725214 (620h34m6.816808249s) [noise=true , address=45.91.116.6:4975 , piece_id=XMLZC7GCIQXV7YV32HKDBKBDODOVMBCORVKEV7SZC4POWLIQNMOQ, seg_pos={0 0}, upload=543793]: Uploading piece... Dec 31 12:03:06.947935113 (620h34m6.819018158s) [noise=true , address=77.90.31.11:4555 , piece_id=NN2EBC66KECKQIXTKB2ZVSM2YVLKLXCCLQWMP5RN2C5UWZM5KFFQ, seg_pos={0 0}, upload=543793]: Uploading piece... Dec 31 12:03:08.633787608 (620h34m8.504870663s) [upload=543793]: Next returned nil segment Dec 31 12:03:08.633836921 (620h34m8.504919967s) [upload=543793]: Waiting for error group managing segments... Dec 31 12:03:08.914501466 (620h34m8.785584501s) [noise=true , address=145.79.227.45:4230 , piece_id=NTHVSF643LXDW5MQ5DQGM3BHPTIXOEL5NEKCLW6ZR7MIJLODF5LA, seg_pos={0 0}, upload=543793]: Done uploading piece. err: <nil> Dec 31 12:03:08.919724370 (620h34m8.790807405s) [noise=true , address=145.79.227.22:4509 , piece_id=YC2PT7M7N7VG5FL4Q2BRDULJDY47HSKSNJ6KWYH5JZGWZTYBFOUA, seg_pos={0 0}, upload=543793]: Done uploading piece. err: <nil> Dec 31 12:03:08.927631316 (620h34m8.798714351s) [noise=true , address=145.79.227.14:4837 , piece_id=LOEPZ4EIZKRWQYII7DUWRJHWBI7MCFHOLJLDZ5EWHPQBRAUXNCOQ, seg_pos={0 0}, upload=543793]: Done uploading piece. err: <nil> Dec 31 12:03:08.932530494 (620h34m8.803613539s) [noise=true , address=145.79.227.36:4517 , piece_id=LGLPG7PMCKGSQ5TRKVQFHZB5DXNMPUKUHFV5AC7CFVACLIEIXQHA, seg_pos={0 0}, upload=543793]: Done uploading piece. err: <nil> Dec 31 12:03:08.937408452 (620h34m8.808491488s) [noise=true , address=145.79.227.37:4566 , piece_id=C4LJ4JIQHTUH72K26O3YTU4CYPOO4XBH7EOMMFPIDB64JB3LZVTA, seg_pos={0 0}, upload=543793]: Done uploading piece. err: <nil> Dec 31 12:03:08.959456975 (620h34m8.83054001s) [noise=true , address=145.79.227.31:4798 , piece_id=WOUKZMWL6E77DEHNOFT65LE2JE77W7GFZ4U4IFTN6SEYVOQ4SBUA, seg_pos={0 0}, upload=543793]: Done uploading piece. err: <nil> Dec 31 12:03:08.959912633 (620h34m8.830995688s) [noise=true , address=145.79.227.8:4329 , piece_id=LXMQABMWVOOCIBKZ2EL7HTW5QTUJGB3FDTZAFELIM7DYZS42WNLQ, seg_pos={0 0}, upload=543793]: Done uploading piece. err: <nil> Dec 31 12:03:08.960449174 (620h34m8.831532219s) [noise=true , address=145.79.227.42:4121 , piece_id=2GLF77AV622O5PLEGOWPZR36PWHLD34W7EC24J242ZMPEMQPS7AQ, seg_pos={0 0}, upload=543793]: Done uploading piece. err: <nil> Dec 31 12:03:08.961014460 (620h34m8.832097495s) [noise=true , address=145.79.227.50:4375 , piece_id=K22R6U4PNLO7ZNOBTKYVJ6URJX5CACIDPQ44DRQYQW4BADMHK4SQ, seg_pos={0 0}, upload=543793]: Done uploading piece. err: <nil> Dec 31 12:03:08.999315790 (620h34m8.870398825s) [noise=true , address=145.79.227.19:4078 , piece_id=K5G5LEONW3RAWGLW3JAZSK7QW6QFHROZO6PX3EF5A3AJKZAXKK6A, seg_pos={0 0}, upload=543793]: Done uploading piece. err: <nil> Dec 31 12:03:09.017297168 (620h34m8.888380204s) [noise=true , address=145.79.227.13:4802 , piece_id=Y35WHD72DB5NBDPIOJVWKMZ2S7MZSKWUJZAIEKVACFML7SZBNKNQ, seg_pos={0 0}, upload=543793]: Done uploading piece. err: <nil> Dec 31 12:03:09.181590309 (620h34m9.052673344s) [noise=true , address=77.90.31.21:4878 , piece_id=6IFN4VOBLW2F3HQBWQC3VAHKU2P4EUQM232KRYBXVDSVQADA5YUA, seg_pos={0 0}, upload=543793]: Done uploading piece. err: <nil> Dec 31 12:03:09.243812086 (620h34m9.114895121s) [noise=true , address=77.90.31.11:4555 , piece_id=NN2EBC66KECKQIXTKB2ZVSM2YVLKLXCCLQWMP5RN2C5UWZM5KFFQ, seg_pos={0 0}, upload=543793]: Done uploading piece. err: <nil> Dec 31 12:03:09.243837224 (620h34m9.114920269s) [seg_pos={0 0}, upload=543793]: Segment reached optimal threshold of 13 pieces. Dec 31 12:03:09.244070898 (620h34m9.115153943s) [noise=true , address=45.91.116.5:4297 , piece_id=32QRF6F4NAVYHWBIOYTLKBD6RJOE3A5DO7N3THJAIPYSBADKLEBQ, seg_pos={0 0}, upload=543793]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 31 12:03:09.244102789 (620h34m9.115185824s) [noise=true , address=45.91.116.5:4297 , piece_id=32QRF6F4NAVYHWBIOYTLKBD6RJOE3A5DO7N3THJAIPYSBADKLEBQ, seg_pos={0 0}, upload=543793]: Long tail cancellation due to optimal threshold Dec 31 12:03:09.244379997 (620h34m9.115463032s) [noise=true , address=77.90.31.16:4736 , piece_id=67UHO2QV3YFPWCDJJPSCHFDEUHFZU4O4YJCAK2KXYE2HMOKNQGZA, seg_pos={0 0}, upload=543793]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 31 12:03:09.244387681 (620h34m9.115470727s) [noise=true , address=77.90.31.16:4736 , piece_id=67UHO2QV3YFPWCDJJPSCHFDEUHFZU4O4YJCAK2KXYE2HMOKNQGZA, seg_pos={0 0}, upload=543793]: Long tail cancellation due to optimal threshold Dec 31 12:03:09.244536826 (620h34m9.115619861s) [noise=true , address=45.91.116.6:4975 , piece_id=XMLZC7GCIQXV7YV32HKDBKBDODOVMBCORVKEV7SZC4POWLIQNMOQ, seg_pos={0 0}, upload=543793]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 31 12:03:09.244545593 (620h34m9.115628638s) [noise=true , address=45.91.116.6:4975 , piece_id=XMLZC7GCIQXV7YV32HKDBKBDODOVMBCORVKEV7SZC4POWLIQNMOQ, seg_pos={0 0}, upload=543793]: Long tail cancellation due to optimal threshold Dec 31 12:03:09.244565681 (620h34m9.115648716s) [seg_pos={0 0}, upload=543793]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 12:03:09.244644411 (620h34m9.115727456s) [upload=543793]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 12:03:09.244650142 (620h34m9.115733187s) [upload=543793]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 12:03:09.534801749 (620h34m9.405884784s) [upload=543793]: Done uploading segments... Dec 31 12:03:09.534822359 (620h34m9.405905404s) [upload=543793]: Upload finished. err: <nil> Duration: 13.324306544s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [..X....X................X..........X.X......X.............X.] duration: 3.908664895s trace: https://jaeger.dev.storj.io/trace/2c96062d1b5b5a44 FAIL (2025-12-31 12:03:46 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.908684672s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [X.X.........X..X..XX....X....X.....XX.XX.X....XX.....X.....X] duration: 30.00292865s trace: https://jaeger.dev.storj.io/trace/2aeaacdc7b2ec FAIL (2025-12-31 12:03:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 31 12:03:06.696854561 (620h34m6.567937596s) [upload=543793]: Starting upload Dec 31 12:03:06.696906339 (620h34m6.567989384s) [upload=543793]: Done starting upload Dec 31 12:03:06.697022900 (620h34m6.568105945s) [upload=543793]: Uploading segments... Dec 31 12:03:06.697101751 (620h34m6.568184786s) [upload=543793]: Got next segment. Inline: false Dec 31 12:03:06.697136497 (620h34m6.568219542s) [upload=543793]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 12:03:06.697142539 (620h34m6.568225584s) [upload=543793]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 12:03:06.941461997 (620h34m6.812545032s) [seg_pos={0 0}, upload=543793]: Begin upload segment... Dec 31 12:03:06.941527512 (620h34m6.812610557s) [seg_pos={0 0}, upload=543793]: Done begin upload segment. Dec 31 12:03:06.941595150 (620h34m6.812678185s) [noise=true , address=77.90.31.21:4878 , piece_id=6IFN4VOBLW2F3HQBWQC3VAHKU2P4EUQM232KRYBXVDSVQADA5YUA, seg_pos={0 0}, upload=543793]: Uploading piece... Dec 31 12:03:06.941627272 (620h34m6.812710317s) [noise=true , address=145.79.227.31:4798 , piece_id=WOUKZMWL6E77DEHNOFT65LE2JE77W7GFZ4U4IFTN6SEYVOQ4SBUA, seg_pos={0 0}, upload=543793]: Uploading piece... Dec 31 12:03:06.941722613 (620h34m6.812805649s) [noise=true , address=145.79.227.37:4566 , piece_id=C4LJ4JIQHTUH72K26O3YTU4CYPOO4XBH7EOMMFPIDB64JB3LZVTA, seg_pos={0 0}, upload=543793]: Uploading piece... Dec 31 12:03:06.942242333 (620h34m6.813325378s) [noise=true , address=145.79.227.8:4329 , piece_id=LXMQABMWVOOCIBKZ2EL7HTW5QTUJGB3FDTZAFELIM7DYZS42WNLQ, seg_pos={0 0}, upload=543793]: Uploading piece... Dec 31 12:03:06.942409090 (620h34m6.813492125s) [noise=true , address=45.91.116.5:4297 , piece_id=32QRF6F4NAVYHWBIOYTLKBD6RJOE3A5DO7N3THJAIPYSBADKLEBQ, seg_pos={0 0}, upload=543793]: Uploading piece... Dec 31 12:03:06.942600695 (620h34m6.81368373s) [noise=true , address=145.79.227.14:4837 , piece_id=LOEPZ4EIZKRWQYII7DUWRJHWBI7MCFHOLJLDZ5EWHPQBRAUXNCOQ, seg_pos={0 0}, upload=543793]: Uploading piece... Dec 31 12:03:06.942760429 (620h34m6.813843464s) [noise=true , address=145.79.227.45:4230 , piece_id=NTHVSF643LXDW5MQ5DQGM3BHPTIXOEL5NEKCLW6ZR7MIJLODF5LA, seg_pos={0 0}, upload=543793]: Uploading piece... Dec 31 12:03:06.943142737 (620h34m6.814225772s) [noise=true , address=145.79.227.36:4517 , piece_id=LGLPG7PMCKGSQ5TRKVQFHZB5DXNMPUKUHFV5AC7CFVACLIEIXQHA, seg_pos={0 0}, upload=543793]: Uploading piece... Dec 31 12:03:06.943205877 (620h34m6.814288912s) [noise=true , address=145.79.227.19:4078 , piece_id=K5G5LEONW3RAWGLW3JAZSK7QW6QFHROZO6PX3EF5A3AJKZAXKK6A, seg_pos={0 0}, upload=543793]: Uploading piece... Dec 31 12:03:06.943520025 (620h34m6.81460307s) [noise=true , address=145.79.227.50:4375 , piece_id=K22R6U4PNLO7ZNOBTKYVJ6URJX5CACIDPQ44DRQYQW4BADMHK4SQ, seg_pos={0 0}, upload=543793]: Uploading piece... Dec 31 12:03:06.943967728 (620h34m6.815050773s) [noise=true , address=145.79.227.13:4802 , piece_id=Y35WHD72DB5NBDPIOJVWKMZ2S7MZSKWUJZAIEKVACFML7SZBNKNQ, seg_pos={0 0}, upload=543793]: Uploading piece... Dec 31 12:03:06.944351178 (620h34m6.815434223s) [noise=true , address=77.90.31.16:4736 , piece_id=67UHO2QV3YFPWCDJJPSCHFDEUHFZU4O4YJCAK2KXYE2HMOKNQGZA, seg_pos={0 0}, upload=543793]: Uploading piece... Dec 31 12:03:06.944709330 (620h34m6.815792365s) [noise=true , address=145.79.227.22:4509 , piece_id=YC2PT7M7N7VG5FL4Q2BRDULJDY47HSKSNJ6KWYH5JZGWZTYBFOUA, seg_pos={0 0}, upload=543793]: Uploading piece... Dec 31 12:03:06.945098561 (620h34m6.816181596s) [noise=true , address=145.79.227.42:4121 , piece_id=2GLF77AV622O5PLEGOWPZR36PWHLD34W7EC24J242ZMPEMQPS7AQ, seg_pos={0 0}, upload=543793]: Uploading piece... Dec 31 12:03:06.945725214 (620h34m6.816808249s) [noise=true , address=45.91.116.6:4975 , piece_id=XMLZC7GCIQXV7YV32HKDBKBDODOVMBCORVKEV7SZC4POWLIQNMOQ, seg_pos={0 0}, upload=543793]: Uploading piece... Dec 31 12:03:06.947935113 (620h34m6.819018158s) [noise=true , address=77.90.31.11:4555 , piece_id=NN2EBC66KECKQIXTKB2ZVSM2YVLKLXCCLQWMP5RN2C5UWZM5KFFQ, seg_pos={0 0}, upload=543793]: Uploading piece... Dec 31 12:03:08.633787608 (620h34m8.504870663s) [upload=543793]: Next returned nil segment Dec 31 12:03:08.633836921 (620h34m8.504919967s) [upload=543793]: Waiting for error group managing segments... Dec 31 12:03:08.914501466 (620h34m8.785584501s) [noise=true , address=145.79.227.45:4230 , piece_id=NTHVSF643LXDW5MQ5DQGM3BHPTIXOEL5NEKCLW6ZR7MIJLODF5LA, seg_pos={0 0}, upload=543793]: Done uploading piece. err: <nil> Dec 31 12:03:08.919724370 (620h34m8.790807405s) [noise=true , address=145.79.227.22:4509 , piece_id=YC2PT7M7N7VG5FL4Q2BRDULJDY47HSKSNJ6KWYH5JZGWZTYBFOUA, seg_pos={0 0}, upload=543793]: Done uploading piece. err: <nil> Dec 31 12:03:08.927631316 (620h34m8.798714351s) [noise=true , address=145.79.227.14:4837 , piece_id=LOEPZ4EIZKRWQYII7DUWRJHWBI7MCFHOLJLDZ5EWHPQBRAUXNCOQ, seg_pos={0 0}, upload=543793]: Done uploading piece. err: <nil> Dec 31 12:03:08.932530494 (620h34m8.803613539s) [noise=true , address=145.79.227.36:4517 , piece_id=LGLPG7PMCKGSQ5TRKVQFHZB5DXNMPUKUHFV5AC7CFVACLIEIXQHA, seg_pos={0 0}, upload=543793]: Done uploading piece. err: <nil> Dec 31 12:03:08.937408452 (620h34m8.808491488s) [noise=true , address=145.79.227.37:4566 , piece_id=C4LJ4JIQHTUH72K26O3YTU4CYPOO4XBH7EOMMFPIDB64JB3LZVTA, seg_pos={0 0}, upload=543793]: Done uploading piece. err: <nil> Dec 31 12:03:08.959456975 (620h34m8.83054001s) [noise=true , address=145.79.227.31:4798 , piece_id=WOUKZMWL6E77DEHNOFT65LE2JE77W7GFZ4U4IFTN6SEYVOQ4SBUA, seg_pos={0 0}, upload=543793]: Done uploading piece. err: <nil> Dec 31 12:03:08.959912633 (620h34m8.830995688s) [noise=true , address=145.79.227.8:4329 , piece_id=LXMQABMWVOOCIBKZ2EL7HTW5QTUJGB3FDTZAFELIM7DYZS42WNLQ, seg_pos={0 0}, upload=543793]: Done uploading piece. err: <nil> Dec 31 12:03:08.960449174 (620h34m8.831532219s) [noise=true , address=145.79.227.42:4121 , piece_id=2GLF77AV622O5PLEGOWPZR36PWHLD34W7EC24J242ZMPEMQPS7AQ, seg_pos={0 0}, upload=543793]: Done uploading piece. err: <nil> Dec 31 12:03:08.961014460 (620h34m8.832097495s) [noise=true , address=145.79.227.50:4375 , piece_id=K22R6U4PNLO7ZNOBTKYVJ6URJX5CACIDPQ44DRQYQW4BADMHK4SQ, seg_pos={0 0}, upload=543793]: Done uploading piece. err: <nil> Dec 31 12:03:08.999315790 (620h34m8.870398825s) [noise=true , address=145.79.227.19:4078 , piece_id=K5G5LEONW3RAWGLW3JAZSK7QW6QFHROZO6PX3EF5A3AJKZAXKK6A, seg_pos={0 0}, upload=543793]: Done uploading piece. err: <nil> Dec 31 12:03:09.017297168 (620h34m8.888380204s) [noise=true , address=145.79.227.13:4802 , piece_id=Y35WHD72DB5NBDPIOJVWKMZ2S7MZSKWUJZAIEKVACFML7SZBNKNQ, seg_pos={0 0}, upload=543793]: Done uploading piece. err: <nil> Dec 31 12:03:09.181590309 (620h34m9.052673344s) [noise=true , address=77.90.31.21:4878 , piece_id=6IFN4VOBLW2F3HQBWQC3VAHKU2P4EUQM232KRYBXVDSVQADA5YUA, seg_pos={0 0}, upload=543793]: Done uploading piece. err: <nil> Dec 31 12:03:09.243812086 (620h34m9.114895121s) [noise=true , address=77.90.31.11:4555 , piece_id=NN2EBC66KECKQIXTKB2ZVSM2YVLKLXCCLQWMP5RN2C5UWZM5KFFQ, seg_pos={0 0}, upload=543793]: Done uploading piece. err: <nil> Dec 31 12:03:09.243837224 (620h34m9.114920269s) [seg_pos={0 0}, upload=543793]: Segment reached optimal threshold of 13 pieces. Dec 31 12:03:09.244070898 (620h34m9.115153943s) [noise=true , address=45.91.116.5:4297 , piece_id=32QRF6F4NAVYHWBIOYTLKBD6RJOE3A5DO7N3THJAIPYSBADKLEBQ, seg_pos={0 0}, upload=543793]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 31 12:03:09.244102789 (620h34m9.115185824s) [noise=true , address=45.91.116.5:4297 , piece_id=32QRF6F4NAVYHWBIOYTLKBD6RJOE3A5DO7N3THJAIPYSBADKLEBQ, seg_pos={0 0}, upload=543793]: Long tail cancellation due to optimal threshold Dec 31 12:03:09.244379997 (620h34m9.115463032s) [noise=true , address=77.90.31.16:4736 , piece_id=67UHO2QV3YFPWCDJJPSCHFDEUHFZU4O4YJCAK2KXYE2HMOKNQGZA, seg_pos={0 0}, upload=543793]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 31 12:03:09.244387681 (620h34m9.115470727s) [noise=true , address=77.90.31.16:4736 , piece_id=67UHO2QV3YFPWCDJJPSCHFDEUHFZU4O4YJCAK2KXYE2HMOKNQGZA, seg_pos={0 0}, upload=543793]: Long tail cancellation due to optimal threshold Dec 31 12:03:09.244536826 (620h34m9.115619861s) [noise=true , address=45.91.116.6:4975 , piece_id=XMLZC7GCIQXV7YV32HKDBKBDODOVMBCORVKEV7SZC4POWLIQNMOQ, seg_pos={0 0}, upload=543793]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 31 12:03:09.244545593 (620h34m9.115628638s) [noise=true , address=45.91.116.6:4975 , piece_id=XMLZC7GCIQXV7YV32HKDBKBDODOVMBCORVKEV7SZC4POWLIQNMOQ, seg_pos={0 0}, upload=543793]: Long tail cancellation due to optimal threshold Dec 31 12:03:09.244565681 (620h34m9.115648716s) [seg_pos={0 0}, upload=543793]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 12:03:09.244644411 (620h34m9.115727456s) [upload=543793]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 12:03:09.244650142 (620h34m9.115733187s) [upload=543793]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 12:03:09.534801749 (620h34m9.405884784s) [upload=543793]: Done uploading segments... Dec 31 12:03:09.534822359 (620h34m9.405905404s) [upload=543793]: Upload finished. err: <nil> Duration: 30.002953267s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [...X....X................X..........X.X......X.............X] duration: 5.001416905s trace: https://jaeger.dev.storj.io/trace/2c96062d1b5b5a44 FAIL (2025-12-31 12:03:27 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00144025s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....X..............................................X.......] duration: 33.8188688s trace: https://jaeger.dev.storj.io/trace/3696061a2156809c FAIL (2025-12-31 12:03:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== .17:4794 , piece_id=ZCNDE7ID7HX43BDZ7KFQKZ6R5V5KUPZWCIRRUQNK65I73JMZQ2JQ, seg_pos={0 0}, upload=543695]: Uploading piece... Dec 31 11:56:12.262721887 (620h27m12.133804932s) [noise=true , address=193.149.164.143:4695 , piece_id=5CJUMEUXTJVQOCSHQKZD4ZHQBTFSMUPIQCI7QMJX2URBSUAIGIFA, seg_pos={0 0}, upload=543695]: Uploading piece... Dec 31 11:56:12.263130455 (620h27m12.1342135s) [noise=true , address=45.91.116.5:4326 , piece_id=VDLNHILWQD6ZTD7JMHQY5BDH2A4IHSJTVHJGDI4IK3OBDZSLOADA, seg_pos={0 0}, upload=543695]: Uploading piece... Dec 31 11:56:12.264175605 (620h27m12.13525865s) [noise=true , address=77.90.31.18:4812 , piece_id=KOGGG55EKOFTTT7OOZ5AULZHU2POHF5DMMSX2RM5M65FC3Y3AXRQ, seg_pos={0 0}, upload=543695]: Uploading piece... Dec 31 11:56:12.264193529 (620h27m12.135276574s) [noise=true , address=145.79.227.49:4358 , piece_id=RP2Y6L6CZD3XOL7XOU3HRDSYZZ6AENXT7LM6XVHZPIZAHSL26BPQ, seg_pos={0 0}, upload=543695]: Uploading piece... Dec 31 11:56:12.264609000 (620h27m12.135692035s) [noise=true , address=45.91.116.11:4145 , piece_id=ZX5FCIOENG6476SS6RDITOCPPCDBXAYOHAOAP4Q545FEBKSSQDDA, seg_pos={0 0}, upload=543695]: Uploading piece... Dec 31 11:56:12.264666079 (620h27m12.135749114s) [noise=true , address=64.49.13.140:4505 , piece_id=YRQ2VLRASSQVB65ONBLZXVLXY7COOSYVUUNMKJO5AOAJKFFIEW2A, seg_pos={0 0}, upload=543695]: Uploading piece... Dec 31 11:56:12.265310566 (620h27m12.136393611s) [noise=true , address=145.79.227.44:4206 , piece_id=TR747XVZACWUD6WGRSROIO5OHFLVBM3XT3XTEWA74GXNXMMYMQHQ, seg_pos={0 0}, upload=543695]: Uploading piece... Dec 31 11:56:12.265745504 (620h27m12.136828549s) [noise=true , address=145.79.227.33:4894 , piece_id=TFZVFOU3GA45OWXOWVA4R2A3W47CS6DCK5K4DAMYYMIH4J5CD23A, seg_pos={0 0}, upload=543695]: Uploading piece... Dec 31 11:56:12.266321861 (620h27m12.137404896s) [noise=true , address=45.91.116.7:4013 , piece_id=A4VDKYMR74JC325EFEP22V3TOGZRMTQB4UOE5I2Y7JKQF32AIZAA, seg_pos={0 0}, upload=543695]: Uploading piece... Dec 31 11:56:12.266868933 (620h27m12.137951978s) [noise=true , address=64.49.13.169:4707 , piece_id=ZQTZNZZQFGHXU2DIVBDUMJTYRAWAPRATLOGQ6AY4A6N45276NDXQ, seg_pos={0 0}, upload=543695]: Uploading piece... Dec 31 11:56:12.267258635 (620h27m12.13834167s) [noise=true , address=77.90.31.13:4657 , piece_id=KSNYKBDG4YOLLPTKC4OFSQR6XOBOGNA64PC3BCKQLRLYGCWFCUUQ, seg_pos={0 0}, upload=543695]: Uploading piece... Dec 31 11:56:12.267587522 (620h27m12.138670567s) [noise=true , address=145.79.227.40:4058 , piece_id=F3RLQOLC5C6JRKLBZICBMGRSJENDRTTP3PGJEFAOBXNHZP4QI3IA, seg_pos={0 0}, upload=543695]: Uploading piece... Dec 31 11:56:12.268440155 (620h27m12.13952319s) [noise=true , address=145.79.227.34:4939 , piece_id=TSCIX7EBMZ72GAOPGWHMCUW4PVEONBZ56A3ODVMIMY6LZZW2XDAQ, seg_pos={0 0}, upload=543695]: Uploading piece... Dec 31 11:56:12.269297889 (620h27m12.140380924s) [noise=true , address=145.79.227.2:4134 , piece_id=HWWXLVVLSMPGUKIP5A5T5IXAB3ZCABH7OA5DIJSNDXRQU23EIFNA, seg_pos={0 0}, upload=543695]: Uploading piece... Dec 31 11:56:13.698835153 (620h27m13.569918188s) [upload=543695]: Next returned nil segment Dec 31 11:56:13.698965801 (620h27m13.570048846s) [upload=543695]: Waiting for error group managing segments... Dec 31 11:56:14.047217086 (620h27m13.918300131s) [noise=true , address=64.49.13.140:4505 , piece_id=YRQ2VLRASSQVB65ONBLZXVLXY7COOSYVUUNMKJO5AOAJKFFIEW2A, seg_pos={0 0}, upload=543695]: Done uploading piece. err: <nil> Dec 31 11:56:14.137538488 (620h27m14.008621523s) [noise=true , address=145.79.227.2:4134 , piece_id=HWWXLVVLSMPGUKIP5A5T5IXAB3ZCABH7OA5DIJSNDXRQU23EIFNA, seg_pos={0 0}, upload=543695]: Done uploading piece. err: <nil> Dec 31 11:56:14.139917508 (620h27m14.011000543s) [noise=true , address=145.79.227.46:4619 , piece_id=BV3NLVQZM457CG2ZYJYZANIDKDVYCQWC64FZGRXFFDV6FFLC2SJA, seg_pos={0 0}, upload=543695]: Done uploading piece. err: <nil> Dec 31 11:56:14.472160101 (620h27m14.343243136s) [noise=true , address=77.90.31.17:4794 , piece_id=ZCNDE7ID7HX43BDZ7KFQKZ6R5V5KUPZWCIRRUQNK65I73JMZQ2JQ, seg_pos={0 0}, upload=543695]: Done uploading piece. err: <nil> Dec 31 11:56:14.476987874 (620h27m14.348070919s) [noise=true , address=45.91.116.5:4326 , piece_id=VDLNHILWQD6ZTD7JMHQY5BDH2A4IHSJTVHJGDI4IK3OBDZSLOADA, seg_pos={0 0}, upload=543695]: Done uploading piece. err: <nil> Dec 31 11:56:14.518004714 (620h27m14.389087749s) [noise=true , address=45.91.116.7:4013 , piece_id=A4VDKYMR74JC325EFEP22V3TOGZRMTQB4UOE5I2Y7JKQF32AIZAA, seg_pos={0 0}, upload=543695]: Done uploading piece. err: <nil> Dec 31 11:56:14.540552146 (620h27m14.411635191s) [noise=true , address=77.90.31.18:4812 , piece_id=KOGGG55EKOFTTT7OOZ5AULZHU2POHF5DMMSX2RM5M65FC3Y3AXRQ, seg_pos={0 0}, upload=543695]: Done uploading piece. err: <nil> Dec 31 11:56:14.677369210 (620h27m14.548452255s) [noise=true , address=145.79.227.40:4058 , piece_id=F3RLQOLC5C6JRKLBZICBMGRSJENDRTTP3PGJEFAOBXNHZP4QI3IA, seg_pos={0 0}, upload=543695]: Done uploading piece. err: <nil> Dec 31 11:56:15.034375707 (620h27m14.905458742s) [noise=true , address=145.79.227.44:4206 , piece_id=TR747XVZACWUD6WGRSROIO5OHFLVBM3XT3XTEWA74GXNXMMYMQHQ, seg_pos={0 0}, upload=543695]: Done uploading piece. err: <nil> Dec 31 11:56:15.095619932 (620h27m14.966702967s) [noise=true , address=145.79.227.49:4358 , piece_id=RP2Y6L6CZD3XOL7XOU3HRDSYZZ6AENXT7LM6XVHZPIZAHSL26BPQ, seg_pos={0 0}, upload=543695]: Done uploading piece. err: <nil> Dec 31 11:56:15.099851950 (620h27m14.970934995s) [noise=true , address=145.79.227.43:4164 , piece_id=QQ4D2S4XAMIDBDSZNYM2G5WT4V7DR4E5WHDSYVT4SZ4YHANAY4QA, seg_pos={0 0}, upload=543695]: Done uploading piece. err: <nil> Dec 31 11:56:15.113070049 (620h27m14.984153094s) [noise=true , address=145.79.227.33:4894 , piece_id=TFZVFOU3GA45OWXOWVA4R2A3W47CS6DCK5K4DAMYYMIH4J5CD23A, seg_pos={0 0}, upload=543695]: Done uploading piece. err: <nil> Dec 31 11:56:15.127931066 (620h27m14.999014101s) [noise=true , address=145.79.227.30:4792 , piece_id=B72NRZMLRWZQRV222OBU6EPXZYYF4K6PXT6L77XZKNVZOA25NLIQ, seg_pos={0 0}, upload=543695]: Done uploading piece. err: <nil> Dec 31 11:56:15.147826669 (620h27m15.018909714s) [noise=true , address=193.149.164.143:4695 , piece_id=5CJUMEUXTJVQOCSHQKZD4ZHQBTFSMUPIQCI7QMJX2URBSUAIGIFA, seg_pos={0 0}, upload=543695]: Done uploading piece. err: <nil> Dec 31 11:56:15.147845695 (620h27m15.01892874s) [seg_pos={0 0}, upload=543695]: Segment reached optimal threshold of 14 pieces. Dec 31 11:56:15.148175553 (620h27m15.019258598s) [noise=true , address=145.79.227.34:4939 , piece_id=TSCIX7EBMZ72GAOPGWHMCUW4PVEONBZ56A3ODVMIMY6LZZW2XDAQ, seg_pos={0 0}, upload=543695]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1h7xrPMs5kHtjm7hiB4hfDaZUHFgtJNoMGqdy3HFif2Aned8eu): protocol: expected piece hash; context canceled; EOF Dec 31 11:56:15.148194439 (620h27m15.019277484s) [noise=true , address=145.79.227.34:4939 , piece_id=TSCIX7EBMZ72GAOPGWHMCUW4PVEONBZ56A3ODVMIMY6LZZW2XDAQ, seg_pos={0 0}, upload=543695]: Long tail cancellation due to optimal threshold Dec 31 11:56:15.148491545 (620h27m15.01957458s) [noise=true , address=77.90.31.13:4657 , piece_id=KSNYKBDG4YOLLPTKC4OFSQR6XOBOGNA64PC3BCKQLRLYGCWFCUUQ, seg_pos={0 0}, upload=543695]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1LLSnWuMa6W7y2jhFUqXoSPEsTyrFsKi8FQDNXF8Tv7XHbP6yt): protocol: write tcp 188.166.225.170:49614->77.90.31.13:4657: use of closed network connection; write tcp 188.166.225.170:49614->77.90.31.13:4657: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49614->77.90.31.13:4657: use of closed network connection Dec 31 11:56:15.148498648 (620h27m15.019581693s) [noise=true , address=77.90.31.13:4657 , piece_id=KSNYKBDG4YOLLPTKC4OFSQR6XOBOGNA64PC3BCKQLRLYGCWFCUUQ, seg_pos={0 0}, upload=543695]: Long tail cancellation due to optimal threshold Dec 31 11:56:15.148641420 (620h27m15.019724465s) [noise=true , address=45.91.116.11:4145 , piece_id=ZX5FCIOENG6476SS6RDITOCPPCDBXAYOHAOAP4Q545FEBKSSQDDA, seg_pos={0 0}, upload=543695]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WKV8PJxq5Kyb3ubCbrfbyYaX6fvRkPMKW2ssUteMFNHm9fP4E): protocol: expected piece hash; context canceled; EOF Dec 31 11:56:15.148647441 (620h27m15.019730476s) [noise=true , address=45.91.116.11:4145 , piece_id=ZX5FCIOENG6476SS6RDITOCPPCDBXAYOHAOAP4Q545FEBKSSQDDA, seg_pos={0 0}, upload=543695]: Long tail cancellation due to optimal threshold Dec 31 11:56:15.148750738 (620h27m15.019833773s) [noise=true , address=64.49.13.169:4707 , piece_id=ZQTZNZZQFGHXU2DIVBDUMJTYRAWAPRATLOGQ6AY4A6N45276NDXQ, seg_pos={0 0}, upload=543695]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:45002->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:45002->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45002->64.49.13.169:4707: use of closed network connection Dec 31 11:56:15.148766589 (620h27m15.019849634s) [noise=true , address=64.49.13.169:4707 , piece_id=ZQTZNZZQFGHXU2DIVBDUMJTYRAWAPRATLOGQ6AY4A6N45276NDXQ, seg_pos={0 0}, upload=543695]: Long tail cancellation due to optimal threshold Dec 31 11:56:15.148782148 (620h27m15.019865193s) [seg_pos={0 0}, upload=543695]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 11:56:15.148845759 (620h27m15.019928804s) [upload=543695]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 11:56:15.148850428 (620h27m15.019933463s) [upload=543695]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 11:56:15.442319734 (620h27m15.313402779s) [upload=543695]: Done uploading segments... Dec 31 11:56:15.442337046 (620h27m15.313420081s) [upload=543695]: Upload finished. err: <nil> Duration: 33.818898825s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [....X....X................X..........X.X......X.............] duration: 3.519866632s trace: https://jaeger.dev.storj.io/trace/66084b8da1020848 FAIL (2025-12-31 12:02:52 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.519892721s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................................X...X.............] duration: 22.598409573s trace: https://jaeger.dev.storj.io/trace/2e598bd65e5b1d02 FAIL (2025-12-31 12:02:46 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 22.598431966s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [X....X....X................X..........X.X......X............] duration: 3.388491717s trace: https://jaeger.dev.storj.io/trace/66084b8da1020848 FAIL (2025-12-31 12:02:28 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.388527726s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [......X..............................................X......] duration: 27.39944806s trace: https://jaeger.dev.storj.io/trace/3696061a2156809c FAIL (2025-12-31 12:02:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== .17:4794 , piece_id=ZCNDE7ID7HX43BDZ7KFQKZ6R5V5KUPZWCIRRUQNK65I73JMZQ2JQ, seg_pos={0 0}, upload=543695]: Uploading piece... Dec 31 11:56:12.262721887 (620h27m12.133804932s) [noise=true , address=193.149.164.143:4695 , piece_id=5CJUMEUXTJVQOCSHQKZD4ZHQBTFSMUPIQCI7QMJX2URBSUAIGIFA, seg_pos={0 0}, upload=543695]: Uploading piece... Dec 31 11:56:12.263130455 (620h27m12.1342135s) [noise=true , address=45.91.116.5:4326 , piece_id=VDLNHILWQD6ZTD7JMHQY5BDH2A4IHSJTVHJGDI4IK3OBDZSLOADA, seg_pos={0 0}, upload=543695]: Uploading piece... Dec 31 11:56:12.264175605 (620h27m12.13525865s) [noise=true , address=77.90.31.18:4812 , piece_id=KOGGG55EKOFTTT7OOZ5AULZHU2POHF5DMMSX2RM5M65FC3Y3AXRQ, seg_pos={0 0}, upload=543695]: Uploading piece... Dec 31 11:56:12.264193529 (620h27m12.135276574s) [noise=true , address=145.79.227.49:4358 , piece_id=RP2Y6L6CZD3XOL7XOU3HRDSYZZ6AENXT7LM6XVHZPIZAHSL26BPQ, seg_pos={0 0}, upload=543695]: Uploading piece... Dec 31 11:56:12.264609000 (620h27m12.135692035s) [noise=true , address=45.91.116.11:4145 , piece_id=ZX5FCIOENG6476SS6RDITOCPPCDBXAYOHAOAP4Q545FEBKSSQDDA, seg_pos={0 0}, upload=543695]: Uploading piece... Dec 31 11:56:12.264666079 (620h27m12.135749114s) [noise=true , address=64.49.13.140:4505 , piece_id=YRQ2VLRASSQVB65ONBLZXVLXY7COOSYVUUNMKJO5AOAJKFFIEW2A, seg_pos={0 0}, upload=543695]: Uploading piece... Dec 31 11:56:12.265310566 (620h27m12.136393611s) [noise=true , address=145.79.227.44:4206 , piece_id=TR747XVZACWUD6WGRSROIO5OHFLVBM3XT3XTEWA74GXNXMMYMQHQ, seg_pos={0 0}, upload=543695]: Uploading piece... Dec 31 11:56:12.265745504 (620h27m12.136828549s) [noise=true , address=145.79.227.33:4894 , piece_id=TFZVFOU3GA45OWXOWVA4R2A3W47CS6DCK5K4DAMYYMIH4J5CD23A, seg_pos={0 0}, upload=543695]: Uploading piece... Dec 31 11:56:12.266321861 (620h27m12.137404896s) [noise=true , address=45.91.116.7:4013 , piece_id=A4VDKYMR74JC325EFEP22V3TOGZRMTQB4UOE5I2Y7JKQF32AIZAA, seg_pos={0 0}, upload=543695]: Uploading piece... Dec 31 11:56:12.266868933 (620h27m12.137951978s) [noise=true , address=64.49.13.169:4707 , piece_id=ZQTZNZZQFGHXU2DIVBDUMJTYRAWAPRATLOGQ6AY4A6N45276NDXQ, seg_pos={0 0}, upload=543695]: Uploading piece... Dec 31 11:56:12.267258635 (620h27m12.13834167s) [noise=true , address=77.90.31.13:4657 , piece_id=KSNYKBDG4YOLLPTKC4OFSQR6XOBOGNA64PC3BCKQLRLYGCWFCUUQ, seg_pos={0 0}, upload=543695]: Uploading piece... Dec 31 11:56:12.267587522 (620h27m12.138670567s) [noise=true , address=145.79.227.40:4058 , piece_id=F3RLQOLC5C6JRKLBZICBMGRSJENDRTTP3PGJEFAOBXNHZP4QI3IA, seg_pos={0 0}, upload=543695]: Uploading piece... Dec 31 11:56:12.268440155 (620h27m12.13952319s) [noise=true , address=145.79.227.34:4939 , piece_id=TSCIX7EBMZ72GAOPGWHMCUW4PVEONBZ56A3ODVMIMY6LZZW2XDAQ, seg_pos={0 0}, upload=543695]: Uploading piece... Dec 31 11:56:12.269297889 (620h27m12.140380924s) [noise=true , address=145.79.227.2:4134 , piece_id=HWWXLVVLSMPGUKIP5A5T5IXAB3ZCABH7OA5DIJSNDXRQU23EIFNA, seg_pos={0 0}, upload=543695]: Uploading piece... Dec 31 11:56:13.698835153 (620h27m13.569918188s) [upload=543695]: Next returned nil segment Dec 31 11:56:13.698965801 (620h27m13.570048846s) [upload=543695]: Waiting for error group managing segments... Dec 31 11:56:14.047217086 (620h27m13.918300131s) [noise=true , address=64.49.13.140:4505 , piece_id=YRQ2VLRASSQVB65ONBLZXVLXY7COOSYVUUNMKJO5AOAJKFFIEW2A, seg_pos={0 0}, upload=543695]: Done uploading piece. err: <nil> Dec 31 11:56:14.137538488 (620h27m14.008621523s) [noise=true , address=145.79.227.2:4134 , piece_id=HWWXLVVLSMPGUKIP5A5T5IXAB3ZCABH7OA5DIJSNDXRQU23EIFNA, seg_pos={0 0}, upload=543695]: Done uploading piece. err: <nil> Dec 31 11:56:14.139917508 (620h27m14.011000543s) [noise=true , address=145.79.227.46:4619 , piece_id=BV3NLVQZM457CG2ZYJYZANIDKDVYCQWC64FZGRXFFDV6FFLC2SJA, seg_pos={0 0}, upload=543695]: Done uploading piece. err: <nil> Dec 31 11:56:14.472160101 (620h27m14.343243136s) [noise=true , address=77.90.31.17:4794 , piece_id=ZCNDE7ID7HX43BDZ7KFQKZ6R5V5KUPZWCIRRUQNK65I73JMZQ2JQ, seg_pos={0 0}, upload=543695]: Done uploading piece. err: <nil> Dec 31 11:56:14.476987874 (620h27m14.348070919s) [noise=true , address=45.91.116.5:4326 , piece_id=VDLNHILWQD6ZTD7JMHQY5BDH2A4IHSJTVHJGDI4IK3OBDZSLOADA, seg_pos={0 0}, upload=543695]: Done uploading piece. err: <nil> Dec 31 11:56:14.518004714 (620h27m14.389087749s) [noise=true , address=45.91.116.7:4013 , piece_id=A4VDKYMR74JC325EFEP22V3TOGZRMTQB4UOE5I2Y7JKQF32AIZAA, seg_pos={0 0}, upload=543695]: Done uploading piece. err: <nil> Dec 31 11:56:14.540552146 (620h27m14.411635191s) [noise=true , address=77.90.31.18:4812 , piece_id=KOGGG55EKOFTTT7OOZ5AULZHU2POHF5DMMSX2RM5M65FC3Y3AXRQ, seg_pos={0 0}, upload=543695]: Done uploading piece. err: <nil> Dec 31 11:56:14.677369210 (620h27m14.548452255s) [noise=true , address=145.79.227.40:4058 , piece_id=F3RLQOLC5C6JRKLBZICBMGRSJENDRTTP3PGJEFAOBXNHZP4QI3IA, seg_pos={0 0}, upload=543695]: Done uploading piece. err: <nil> Dec 31 11:56:15.034375707 (620h27m14.905458742s) [noise=true , address=145.79.227.44:4206 , piece_id=TR747XVZACWUD6WGRSROIO5OHFLVBM3XT3XTEWA74GXNXMMYMQHQ, seg_pos={0 0}, upload=543695]: Done uploading piece. err: <nil> Dec 31 11:56:15.095619932 (620h27m14.966702967s) [noise=true , address=145.79.227.49:4358 , piece_id=RP2Y6L6CZD3XOL7XOU3HRDSYZZ6AENXT7LM6XVHZPIZAHSL26BPQ, seg_pos={0 0}, upload=543695]: Done uploading piece. err: <nil> Dec 31 11:56:15.099851950 (620h27m14.970934995s) [noise=true , address=145.79.227.43:4164 , piece_id=QQ4D2S4XAMIDBDSZNYM2G5WT4V7DR4E5WHDSYVT4SZ4YHANAY4QA, seg_pos={0 0}, upload=543695]: Done uploading piece. err: <nil> Dec 31 11:56:15.113070049 (620h27m14.984153094s) [noise=true , address=145.79.227.33:4894 , piece_id=TFZVFOU3GA45OWXOWVA4R2A3W47CS6DCK5K4DAMYYMIH4J5CD23A, seg_pos={0 0}, upload=543695]: Done uploading piece. err: <nil> Dec 31 11:56:15.127931066 (620h27m14.999014101s) [noise=true , address=145.79.227.30:4792 , piece_id=B72NRZMLRWZQRV222OBU6EPXZYYF4K6PXT6L77XZKNVZOA25NLIQ, seg_pos={0 0}, upload=543695]: Done uploading piece. err: <nil> Dec 31 11:56:15.147826669 (620h27m15.018909714s) [noise=true , address=193.149.164.143:4695 , piece_id=5CJUMEUXTJVQOCSHQKZD4ZHQBTFSMUPIQCI7QMJX2URBSUAIGIFA, seg_pos={0 0}, upload=543695]: Done uploading piece. err: <nil> Dec 31 11:56:15.147845695 (620h27m15.01892874s) [seg_pos={0 0}, upload=543695]: Segment reached optimal threshold of 14 pieces. Dec 31 11:56:15.148175553 (620h27m15.019258598s) [noise=true , address=145.79.227.34:4939 , piece_id=TSCIX7EBMZ72GAOPGWHMCUW4PVEONBZ56A3ODVMIMY6LZZW2XDAQ, seg_pos={0 0}, upload=543695]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1h7xrPMs5kHtjm7hiB4hfDaZUHFgtJNoMGqdy3HFif2Aned8eu): protocol: expected piece hash; context canceled; EOF Dec 31 11:56:15.148194439 (620h27m15.019277484s) [noise=true , address=145.79.227.34:4939 , piece_id=TSCIX7EBMZ72GAOPGWHMCUW4PVEONBZ56A3ODVMIMY6LZZW2XDAQ, seg_pos={0 0}, upload=543695]: Long tail cancellation due to optimal threshold Dec 31 11:56:15.148491545 (620h27m15.01957458s) [noise=true , address=77.90.31.13:4657 , piece_id=KSNYKBDG4YOLLPTKC4OFSQR6XOBOGNA64PC3BCKQLRLYGCWFCUUQ, seg_pos={0 0}, upload=543695]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1LLSnWuMa6W7y2jhFUqXoSPEsTyrFsKi8FQDNXF8Tv7XHbP6yt): protocol: write tcp 188.166.225.170:49614->77.90.31.13:4657: use of closed network connection; write tcp 188.166.225.170:49614->77.90.31.13:4657: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49614->77.90.31.13:4657: use of closed network connection Dec 31 11:56:15.148498648 (620h27m15.019581693s) [noise=true , address=77.90.31.13:4657 , piece_id=KSNYKBDG4YOLLPTKC4OFSQR6XOBOGNA64PC3BCKQLRLYGCWFCUUQ, seg_pos={0 0}, upload=543695]: Long tail cancellation due to optimal threshold Dec 31 11:56:15.148641420 (620h27m15.019724465s) [noise=true , address=45.91.116.11:4145 , piece_id=ZX5FCIOENG6476SS6RDITOCPPCDBXAYOHAOAP4Q545FEBKSSQDDA, seg_pos={0 0}, upload=543695]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WKV8PJxq5Kyb3ubCbrfbyYaX6fvRkPMKW2ssUteMFNHm9fP4E): protocol: expected piece hash; context canceled; EOF Dec 31 11:56:15.148647441 (620h27m15.019730476s) [noise=true , address=45.91.116.11:4145 , piece_id=ZX5FCIOENG6476SS6RDITOCPPCDBXAYOHAOAP4Q545FEBKSSQDDA, seg_pos={0 0}, upload=543695]: Long tail cancellation due to optimal threshold Dec 31 11:56:15.148750738 (620h27m15.019833773s) [noise=true , address=64.49.13.169:4707 , piece_id=ZQTZNZZQFGHXU2DIVBDUMJTYRAWAPRATLOGQ6AY4A6N45276NDXQ, seg_pos={0 0}, upload=543695]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:45002->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:45002->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45002->64.49.13.169:4707: use of closed network connection Dec 31 11:56:15.148766589 (620h27m15.019849634s) [noise=true , address=64.49.13.169:4707 , piece_id=ZQTZNZZQFGHXU2DIVBDUMJTYRAWAPRATLOGQ6AY4A6N45276NDXQ, seg_pos={0 0}, upload=543695]: Long tail cancellation due to optimal threshold Dec 31 11:56:15.148782148 (620h27m15.019865193s) [seg_pos={0 0}, upload=543695]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 11:56:15.148845759 (620h27m15.019928804s) [upload=543695]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 11:56:15.148850428 (620h27m15.019933463s) [upload=543695]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 11:56:15.442319734 (620h27m15.313402779s) [upload=543695]: Done uploading segments... Dec 31 11:56:15.442337046 (620h27m15.313420081s) [upload=543695]: Upload finished. err: <nil> Duration: 27.39947899s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [XX.X.........X..X..XX....X....X.....XX.XX.X....XX.....X.....] duration: 6.080678116s trace: https://jaeger.dev.storj.io/trace/14b1d7b8cb45e76f FAIL (2025-12-31 12:02:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, Piece ID: K6JW5TBRCAKD7UZ7NVSS6MNDQSRGEHMDX3SL7UJWCYPMGJXG4M2Q) context deadline exceeded; piecestore: (Node ID: 12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, Piece ID: IXV6CZGSMYNG42OBYCXHXLTZ6PLVBXBRJZPMMY3FYZZWJECR2RKA) context deadline exceeded ======LOGS====== Dec 31 11:58:35.594635425 (620h29m35.46571846s) [upload=543727]: Starting upload Dec 31 11:58:35.594722040 (620h29m35.465805075s) [upload=543727]: Done starting upload Dec 31 11:58:35.594792234 (620h29m35.465875279s) [upload=543727]: Uploading segments... Dec 31 11:58:35.595076655 (620h29m35.46615969s) [upload=543727]: Got next segment. Inline: false Dec 31 11:58:35.595146408 (620h29m35.466229443s) [upload=543727]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 11:58:35.595151949 (620h29m35.466234994s) [upload=543727]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 11:58:35.841221299 (620h29m35.712304334s) [seg_pos={0 0}, upload=543727]: Begin upload segment... Dec 31 11:58:35.841265373 (620h29m35.712348418s) [seg_pos={0 0}, upload=543727]: Done begin upload segment. Dec 31 11:58:35.841290982 (620h29m35.712374017s) [noise=true , address=145.79.227.16:4909 , piece_id=SNFCBFTGFNPGA2OEFCIL5UVW7QY3NEEZPWVMDYB2WTHW74RCK3BA, seg_pos={0 0}, upload=543727]: Uploading piece... Dec 31 11:58:35.845402050 (620h29m35.716485095s) [noise=true , address=77.90.31.14:4663 , piece_id=YOFV3624FNCQ66JZE2ZKHSHVPIYZPODHUTOKDFSQB5IWMBM746IA, seg_pos={0 0}, upload=543727]: Uploading piece... Dec 31 11:58:35.845868028 (620h29m35.716951073s) [noise=true , address=145.79.227.27:4690 , piece_id=4XWT6ARZW57J5PT5JCQRVZZQ7DAJCRV6SU2FA4PTROQA4HO62XYA, seg_pos={0 0}, upload=543727]: Uploading piece... Dec 31 11:58:35.847161631 (620h29m35.718244676s) [noise=true , address=45.91.116.10:4083 , piece_id=DLMF5JTBQBEELTRAQWVX45PGDF6DIW6VUKDXGTVZGLGKFARRGYTA, seg_pos={0 0}, upload=543727]: Uploading piece... Dec 31 11:58:35.847419842 (620h29m35.718502877s) [noise=true , address=77.90.31.11:4554 , piece_id=2FQN56B35YPXHWKGHL5NL66QH2G5E74EOCTAPCMJPCU5AXXAGFWA, seg_pos={0 0}, upload=543727]: Uploading piece... Dec 31 11:58:35.848508435 (620h29m35.71959147s) [noise=true , address=77.90.31.21:4877 , piece_id=JKUUV3N45LOKIZ3VZKIFJKDSY7EDJYZLWCRCZ4LDWTVGA7EOYDJA, seg_pos={0 0}, upload=543727]: Uploading piece... Dec 31 11:58:35.849066748 (620h29m35.720149783s) [noise=true , address=45.91.116.12:4225 , piece_id=IXV6CZGSMYNG42OBYCXHXLTZ6PLVBXBRJZPMMY3FYZZWJECR2RKA, seg_pos={0 0}, upload=543727]: Uploading piece... Dec 31 11:58:35.849929060 (620h29m35.721012095s) [noise=true , address=145.79.227.13:4802 , piece_id=KHPURVADASJLVUJ55E64WXZ6G5Y5T774BZYT52VXU5WR3XMRIGFQ, seg_pos={0 0}, upload=543727]: Uploading piece... Dec 31 11:58:35.850631718 (620h29m35.721714753s) [noise=true , address=145.79.227.47:4265 , piece_id=HJ3CU7G3EJPY4OJJS3US462OOIWYAJL25XWRDUQX63ZMN55KKNCQ, seg_pos={0 0}, upload=543727]: Uploading piece... Dec 31 11:58:35.851263250 (620h29m35.722346296s) [noise=true , address=145.79.227.45:4230 , piece_id=UUHCZZLDHI7M6567AVXN5TP5FLGPLDFNJCN2H6PO6F3GNFDJF3RQ, seg_pos={0 0}, upload=543727]: Uploading piece... Dec 31 11:58:35.852578124 (620h29m35.723661159s) [noise=true , address=77.90.31.10:4519 , piece_id=VNFSOLAGMAJMGYPFXODDT6ZWSN6ZV42UKGZBARQRUVGSREW4ZDEA, seg_pos={0 0}, upload=543727]: Uploading piece... Dec 31 11:58:35.853307623 (620h29m35.724390658s) [noise=true , address=145.79.227.24:4581 , piece_id=U3PYPUOKYBEYW4HSB4SE6LXI5KSWCF4Z4SDDNMKCEKNMIRFJRIVA, seg_pos={0 0}, upload=543727]: Uploading piece... Dec 31 11:58:35.854473573 (620h29m35.725556618s) [noise=true , address=45.91.116.4:4939 , piece_id=YZ2N3RAMAQPT4URGBKGL6ZLUHES2G5JRF5OQSHW4HJBICAKXBSZA, seg_pos={0 0}, upload=543727]: Uploading piece... Dec 31 11:58:35.855788177 (620h29m35.726871212s) [noise=true , address=45.91.116.14:4192 , piece_id=ERQ7YPQVRDKLV2B3XPRI2BWUU5XZ2QIGEL67CEWTBZ66FCNOXOOA, seg_pos={0 0}, upload=543727]: Uploading piece... Dec 31 11:58:35.856163310 (620h29m35.727246355s) [noise=true , address=77.90.31.19:4842 , piece_id=K6JW5TBRCAKD7UZ7NVSS6MNDQSRGEHMDX3SL7UJWCYPMGJXG4M2Q, seg_pos={0 0}, upload=543727]: Uploading piece... Dec 31 11:58:35.857671252 (620h29m35.728754287s) [noise=true , address=145.79.227.49:4337 , piece_id=L5B3IDX2SWVQSJWKZHADBRCYHR6NQETXSWMCVHVEBHNU34RP5KRA, seg_pos={0 0}, upload=543727]: Uploading piece... Dec 31 11:58:37.503302041 (620h29m37.374385076s) [upload=543727]: Next returned nil segment Dec 31 11:58:37.503322560 (620h29m37.374405596s) [upload=543727]: Waiting for error group managing segments... Dec 31 11:58:37.834739542 (620h29m37.705822577s) [noise=true , address=145.79.227.47:4265 , piece_id=HJ3CU7G3EJPY4OJJS3US462OOIWYAJL25XWRDUQX63ZMN55KKNCQ, seg_pos={0 0}, upload=543727]: Done uploading piece. err: <nil> Dec 31 11:58:37.838953555 (620h29m37.7100366s) [noise=true , address=145.79.227.27:4690 , piece_id=4XWT6ARZW57J5PT5JCQRVZZQ7DAJCRV6SU2FA4PTROQA4HO62XYA, seg_pos={0 0}, upload=543727]: Done uploading piece. err: <nil> Dec 31 11:58:37.859738432 (620h29m37.730821477s) [noise=true , address=145.79.227.16:4909 , piece_id=SNFCBFTGFNPGA2OEFCIL5UVW7QY3NEEZPWVMDYB2WTHW74RCK3BA, seg_pos={0 0}, upload=543727]: Done uploading piece. err: <nil> Dec 31 11:58:37.867873061 (620h29m37.738956096s) [noise=true , address=145.79.227.45:4230 , piece_id=UUHCZZLDHI7M6567AVXN5TP5FLGPLDFNJCN2H6PO6F3GNFDJF3RQ, seg_pos={0 0}, upload=543727]: Done uploading piece. err: <nil> Dec 31 11:58:37.920820508 (620h29m37.791903543s) [noise=true , address=145.79.227.13:4802 , piece_id=KHPURVADASJLVUJ55E64WXZ6G5Y5T774BZYT52VXU5WR3XMRIGFQ, seg_pos={0 0}, upload=543727]: Done uploading piece. err: <nil> Dec 31 11:58:37.924887062 (620h29m37.795970107s) [noise=true , address=145.79.227.24:4581 , piece_id=U3PYPUOKYBEYW4HSB4SE6LXI5KSWCF4Z4SDDNMKCEKNMIRFJRIVA, seg_pos={0 0}, upload=543727]: Done uploading piece. err: <nil> Dec 31 11:58:37.940974634 (620h29m37.812057679s) [noise=true , address=145.79.227.49:4337 , piece_id=L5B3IDX2SWVQSJWKZHADBRCYHR6NQETXSWMCVHVEBHNU34RP5KRA, seg_pos={0 0}, upload=543727]: Done uploading piece. err: <nil> Dec 31 11:58:38.003965085 (620h29m37.87504814s) [noise=true , address=77.90.31.11:4554 , piece_id=2FQN56B35YPXHWKGHL5NL66QH2G5E74EOCTAPCMJPCU5AXXAGFWA, seg_pos={0 0}, upload=543727]: Done uploading piece. err: <nil> Dec 31 11:58:38.108808727 (620h29m37.979891762s) [noise=true , address=77.90.31.19:4842 , piece_id=K6JW5TBRCAKD7UZ7NVSS6MNDQSRGEHMDX3SL7UJWCYPMGJXG4M2Q, seg_pos={0 0}, upload=543727]: Done uploading piece. err: <nil> Dec 31 11:58:38.136822929 (620h29m38.007905964s) [noise=true , address=77.90.31.10:4519 , piece_id=VNFSOLAGMAJMGYPFXODDT6ZWSN6ZV42UKGZBARQRUVGSREW4ZDEA, seg_pos={0 0}, upload=543727]: Done uploading piece. err: <nil> Dec 31 11:58:38.173281600 (620h29m38.044364646s) [noise=true , address=45.91.116.12:4225 , piece_id=IXV6CZGSMYNG42OBYCXHXLTZ6PLVBXBRJZPMMY3FYZZWJECR2RKA, seg_pos={0 0}, upload=543727]: Done uploading piece. err: <nil> Dec 31 11:58:38.264543502 (620h29m38.135626547s) [noise=true , address=45.91.116.4:4939 , piece_id=YZ2N3RAMAQPT4URGBKGL6ZLUHES2G5JRF5OQSHW4HJBICAKXBSZA, seg_pos={0 0}, upload=543727]: Done uploading piece. err: <nil> Dec 31 11:58:38.602959671 (620h29m38.474042706s) [noise=true , address=45.91.116.14:4192 , piece_id=ERQ7YPQVRDKLV2B3XPRI2BWUU5XZ2QIGEL67CEWTBZ66FCNOXOOA, seg_pos={0 0}, upload=543727]: Done uploading piece. err: <nil> Dec 31 11:58:38.602985380 (620h29m38.474068415s) [seg_pos={0 0}, upload=543727]: Segment reached optimal threshold of 13 pieces. Dec 31 11:58:38.603240727 (620h29m38.474323772s) [noise=true , address=77.90.31.21:4877 , piece_id=JKUUV3N45LOKIZ3VZKIFJKDSY7EDJYZLWCRCZ4LDWTVGA7EOYDJA, seg_pos={0 0}, upload=543727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Dec 31 11:58:38.603253060 (620h29m38.474336105s) [noise=true , address=77.90.31.21:4877 , piece_id=JKUUV3N45LOKIZ3VZKIFJKDSY7EDJYZLWCRCZ4LDWTVGA7EOYDJA, seg_pos={0 0}, upload=543727]: Long tail cancellation due to optimal threshold Dec 31 11:58:38.603277977 (620h29m38.474361023s) [noise=true , address=45.91.116.10:4083 , piece_id=DLMF5JTBQBEELTRAQWVX45PGDF6DIW6VUKDXGTVZGLGKFARRGYTA, seg_pos={0 0}, upload=543727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Dec 31 11:58:38.603285552 (620h29m38.474368597s) [noise=true , address=45.91.116.10:4083 , piece_id=DLMF5JTBQBEELTRAQWVX45PGDF6DIW6VUKDXGTVZGLGKFARRGYTA, seg_pos={0 0}, upload=543727]: Long tail cancellation due to optimal threshold Dec 31 11:58:38.603539265 (620h29m38.47462231s) [noise=true , address=77.90.31.14:4663 , piece_id=YOFV3624FNCQ66JZE2ZKHSHVPIYZPODHUTOKDFSQB5IWMBM746IA, seg_pos={0 0}, upload=543727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE): protocol: expected piece hash; context canceled; EOF Dec 31 11:58:38.603548722 (620h29m38.474631767s) [noise=true , address=77.90.31.14:4663 , piece_id=YOFV3624FNCQ66JZE2ZKHSHVPIYZPODHUTOKDFSQB5IWMBM746IA, seg_pos={0 0}, upload=543727]: Long tail cancellation due to optimal threshold Dec 31 11:58:38.603570654 (620h29m38.474653689s) [seg_pos={0 0}, upload=543727]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 11:58:38.603665946 (620h29m38.474748991s) [upload=543727]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 11:58:38.603672639 (620h29m38.474755684s) [upload=543727]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 11:58:38.889947435 (620h29m38.76103048s) [upload=543727]: Done uploading segments... Dec 31 11:58:38.889970048 (620h29m38.761053093s) [upload=543727]: Upload finished. err: <nil> Duration: 6.080794446s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [.X....X....X................X..........X.X......X...........] duration: 3.87615438s trace: https://jaeger.dev.storj.io/trace/66084b8da1020848 FAIL (2025-12-31 12:01:38 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.876178856s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [XXX.X.........X..X..XX....X....X.....XX.XX.X....XX.....X....] duration: 6.003990036s trace: https://jaeger.dev.storj.io/trace/14b1d7b8cb45e76f FAIL (2025-12-31 12:01:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, Piece ID: K6JW5TBRCAKD7UZ7NVSS6MNDQSRGEHMDX3SL7UJWCYPMGJXG4M2Q) context deadline exceeded; piecestore: (Node ID: 12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, Piece ID: IXV6CZGSMYNG42OBYCXHXLTZ6PLVBXBRJZPMMY3FYZZWJECR2RKA) context deadline exceeded ======LOGS====== Dec 31 11:58:35.594635425 (620h29m35.46571846s) [upload=543727]: Starting upload Dec 31 11:58:35.594722040 (620h29m35.465805075s) [upload=543727]: Done starting upload Dec 31 11:58:35.594792234 (620h29m35.465875279s) [upload=543727]: Uploading segments... Dec 31 11:58:35.595076655 (620h29m35.46615969s) [upload=543727]: Got next segment. Inline: false Dec 31 11:58:35.595146408 (620h29m35.466229443s) [upload=543727]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 11:58:35.595151949 (620h29m35.466234994s) [upload=543727]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 11:58:35.841221299 (620h29m35.712304334s) [seg_pos={0 0}, upload=543727]: Begin upload segment... Dec 31 11:58:35.841265373 (620h29m35.712348418s) [seg_pos={0 0}, upload=543727]: Done begin upload segment. Dec 31 11:58:35.841290982 (620h29m35.712374017s) [noise=true , address=145.79.227.16:4909 , piece_id=SNFCBFTGFNPGA2OEFCIL5UVW7QY3NEEZPWVMDYB2WTHW74RCK3BA, seg_pos={0 0}, upload=543727]: Uploading piece... Dec 31 11:58:35.845402050 (620h29m35.716485095s) [noise=true , address=77.90.31.14:4663 , piece_id=YOFV3624FNCQ66JZE2ZKHSHVPIYZPODHUTOKDFSQB5IWMBM746IA, seg_pos={0 0}, upload=543727]: Uploading piece... Dec 31 11:58:35.845868028 (620h29m35.716951073s) [noise=true , address=145.79.227.27:4690 , piece_id=4XWT6ARZW57J5PT5JCQRVZZQ7DAJCRV6SU2FA4PTROQA4HO62XYA, seg_pos={0 0}, upload=543727]: Uploading piece... Dec 31 11:58:35.847161631 (620h29m35.718244676s) [noise=true , address=45.91.116.10:4083 , piece_id=DLMF5JTBQBEELTRAQWVX45PGDF6DIW6VUKDXGTVZGLGKFARRGYTA, seg_pos={0 0}, upload=543727]: Uploading piece... Dec 31 11:58:35.847419842 (620h29m35.718502877s) [noise=true , address=77.90.31.11:4554 , piece_id=2FQN56B35YPXHWKGHL5NL66QH2G5E74EOCTAPCMJPCU5AXXAGFWA, seg_pos={0 0}, upload=543727]: Uploading piece... Dec 31 11:58:35.848508435 (620h29m35.71959147s) [noise=true , address=77.90.31.21:4877 , piece_id=JKUUV3N45LOKIZ3VZKIFJKDSY7EDJYZLWCRCZ4LDWTVGA7EOYDJA, seg_pos={0 0}, upload=543727]: Uploading piece... Dec 31 11:58:35.849066748 (620h29m35.720149783s) [noise=true , address=45.91.116.12:4225 , piece_id=IXV6CZGSMYNG42OBYCXHXLTZ6PLVBXBRJZPMMY3FYZZWJECR2RKA, seg_pos={0 0}, upload=543727]: Uploading piece... Dec 31 11:58:35.849929060 (620h29m35.721012095s) [noise=true , address=145.79.227.13:4802 , piece_id=KHPURVADASJLVUJ55E64WXZ6G5Y5T774BZYT52VXU5WR3XMRIGFQ, seg_pos={0 0}, upload=543727]: Uploading piece... Dec 31 11:58:35.850631718 (620h29m35.721714753s) [noise=true , address=145.79.227.47:4265 , piece_id=HJ3CU7G3EJPY4OJJS3US462OOIWYAJL25XWRDUQX63ZMN55KKNCQ, seg_pos={0 0}, upload=543727]: Uploading piece... Dec 31 11:58:35.851263250 (620h29m35.722346296s) [noise=true , address=145.79.227.45:4230 , piece_id=UUHCZZLDHI7M6567AVXN5TP5FLGPLDFNJCN2H6PO6F3GNFDJF3RQ, seg_pos={0 0}, upload=543727]: Uploading piece... Dec 31 11:58:35.852578124 (620h29m35.723661159s) [noise=true , address=77.90.31.10:4519 , piece_id=VNFSOLAGMAJMGYPFXODDT6ZWSN6ZV42UKGZBARQRUVGSREW4ZDEA, seg_pos={0 0}, upload=543727]: Uploading piece... Dec 31 11:58:35.853307623 (620h29m35.724390658s) [noise=true , address=145.79.227.24:4581 , piece_id=U3PYPUOKYBEYW4HSB4SE6LXI5KSWCF4Z4SDDNMKCEKNMIRFJRIVA, seg_pos={0 0}, upload=543727]: Uploading piece... Dec 31 11:58:35.854473573 (620h29m35.725556618s) [noise=true , address=45.91.116.4:4939 , piece_id=YZ2N3RAMAQPT4URGBKGL6ZLUHES2G5JRF5OQSHW4HJBICAKXBSZA, seg_pos={0 0}, upload=543727]: Uploading piece... Dec 31 11:58:35.855788177 (620h29m35.726871212s) [noise=true , address=45.91.116.14:4192 , piece_id=ERQ7YPQVRDKLV2B3XPRI2BWUU5XZ2QIGEL67CEWTBZ66FCNOXOOA, seg_pos={0 0}, upload=543727]: Uploading piece... Dec 31 11:58:35.856163310 (620h29m35.727246355s) [noise=true , address=77.90.31.19:4842 , piece_id=K6JW5TBRCAKD7UZ7NVSS6MNDQSRGEHMDX3SL7UJWCYPMGJXG4M2Q, seg_pos={0 0}, upload=543727]: Uploading piece... Dec 31 11:58:35.857671252 (620h29m35.728754287s) [noise=true , address=145.79.227.49:4337 , piece_id=L5B3IDX2SWVQSJWKZHADBRCYHR6NQETXSWMCVHVEBHNU34RP5KRA, seg_pos={0 0}, upload=543727]: Uploading piece... Dec 31 11:58:37.503302041 (620h29m37.374385076s) [upload=543727]: Next returned nil segment Dec 31 11:58:37.503322560 (620h29m37.374405596s) [upload=543727]: Waiting for error group managing segments... Dec 31 11:58:37.834739542 (620h29m37.705822577s) [noise=true , address=145.79.227.47:4265 , piece_id=HJ3CU7G3EJPY4OJJS3US462OOIWYAJL25XWRDUQX63ZMN55KKNCQ, seg_pos={0 0}, upload=543727]: Done uploading piece. err: <nil> Dec 31 11:58:37.838953555 (620h29m37.7100366s) [noise=true , address=145.79.227.27:4690 , piece_id=4XWT6ARZW57J5PT5JCQRVZZQ7DAJCRV6SU2FA4PTROQA4HO62XYA, seg_pos={0 0}, upload=543727]: Done uploading piece. err: <nil> Dec 31 11:58:37.859738432 (620h29m37.730821477s) [noise=true , address=145.79.227.16:4909 , piece_id=SNFCBFTGFNPGA2OEFCIL5UVW7QY3NEEZPWVMDYB2WTHW74RCK3BA, seg_pos={0 0}, upload=543727]: Done uploading piece. err: <nil> Dec 31 11:58:37.867873061 (620h29m37.738956096s) [noise=true , address=145.79.227.45:4230 , piece_id=UUHCZZLDHI7M6567AVXN5TP5FLGPLDFNJCN2H6PO6F3GNFDJF3RQ, seg_pos={0 0}, upload=543727]: Done uploading piece. err: <nil> Dec 31 11:58:37.920820508 (620h29m37.791903543s) [noise=true , address=145.79.227.13:4802 , piece_id=KHPURVADASJLVUJ55E64WXZ6G5Y5T774BZYT52VXU5WR3XMRIGFQ, seg_pos={0 0}, upload=543727]: Done uploading piece. err: <nil> Dec 31 11:58:37.924887062 (620h29m37.795970107s) [noise=true , address=145.79.227.24:4581 , piece_id=U3PYPUOKYBEYW4HSB4SE6LXI5KSWCF4Z4SDDNMKCEKNMIRFJRIVA, seg_pos={0 0}, upload=543727]: Done uploading piece. err: <nil> Dec 31 11:58:37.940974634 (620h29m37.812057679s) [noise=true , address=145.79.227.49:4337 , piece_id=L5B3IDX2SWVQSJWKZHADBRCYHR6NQETXSWMCVHVEBHNU34RP5KRA, seg_pos={0 0}, upload=543727]: Done uploading piece. err: <nil> Dec 31 11:58:38.003965085 (620h29m37.87504814s) [noise=true , address=77.90.31.11:4554 , piece_id=2FQN56B35YPXHWKGHL5NL66QH2G5E74EOCTAPCMJPCU5AXXAGFWA, seg_pos={0 0}, upload=543727]: Done uploading piece. err: <nil> Dec 31 11:58:38.108808727 (620h29m37.979891762s) [noise=true , address=77.90.31.19:4842 , piece_id=K6JW5TBRCAKD7UZ7NVSS6MNDQSRGEHMDX3SL7UJWCYPMGJXG4M2Q, seg_pos={0 0}, upload=543727]: Done uploading piece. err: <nil> Dec 31 11:58:38.136822929 (620h29m38.007905964s) [noise=true , address=77.90.31.10:4519 , piece_id=VNFSOLAGMAJMGYPFXODDT6ZWSN6ZV42UKGZBARQRUVGSREW4ZDEA, seg_pos={0 0}, upload=543727]: Done uploading piece. err: <nil> Dec 31 11:58:38.173281600 (620h29m38.044364646s) [noise=true , address=45.91.116.12:4225 , piece_id=IXV6CZGSMYNG42OBYCXHXLTZ6PLVBXBRJZPMMY3FYZZWJECR2RKA, seg_pos={0 0}, upload=543727]: Done uploading piece. err: <nil> Dec 31 11:58:38.264543502 (620h29m38.135626547s) [noise=true , address=45.91.116.4:4939 , piece_id=YZ2N3RAMAQPT4URGBKGL6ZLUHES2G5JRF5OQSHW4HJBICAKXBSZA, seg_pos={0 0}, upload=543727]: Done uploading piece. err: <nil> Dec 31 11:58:38.602959671 (620h29m38.474042706s) [noise=true , address=45.91.116.14:4192 , piece_id=ERQ7YPQVRDKLV2B3XPRI2BWUU5XZ2QIGEL67CEWTBZ66FCNOXOOA, seg_pos={0 0}, upload=543727]: Done uploading piece. err: <nil> Dec 31 11:58:38.602985380 (620h29m38.474068415s) [seg_pos={0 0}, upload=543727]: Segment reached optimal threshold of 13 pieces. Dec 31 11:58:38.603240727 (620h29m38.474323772s) [noise=true , address=77.90.31.21:4877 , piece_id=JKUUV3N45LOKIZ3VZKIFJKDSY7EDJYZLWCRCZ4LDWTVGA7EOYDJA, seg_pos={0 0}, upload=543727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Dec 31 11:58:38.603253060 (620h29m38.474336105s) [noise=true , address=77.90.31.21:4877 , piece_id=JKUUV3N45LOKIZ3VZKIFJKDSY7EDJYZLWCRCZ4LDWTVGA7EOYDJA, seg_pos={0 0}, upload=543727]: Long tail cancellation due to optimal threshold Dec 31 11:58:38.603277977 (620h29m38.474361023s) [noise=true , address=45.91.116.10:4083 , piece_id=DLMF5JTBQBEELTRAQWVX45PGDF6DIW6VUKDXGTVZGLGKFARRGYTA, seg_pos={0 0}, upload=543727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Dec 31 11:58:38.603285552 (620h29m38.474368597s) [noise=true , address=45.91.116.10:4083 , piece_id=DLMF5JTBQBEELTRAQWVX45PGDF6DIW6VUKDXGTVZGLGKFARRGYTA, seg_pos={0 0}, upload=543727]: Long tail cancellation due to optimal threshold Dec 31 11:58:38.603539265 (620h29m38.47462231s) [noise=true , address=77.90.31.14:4663 , piece_id=YOFV3624FNCQ66JZE2ZKHSHVPIYZPODHUTOKDFSQB5IWMBM746IA, seg_pos={0 0}, upload=543727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE): protocol: expected piece hash; context canceled; EOF Dec 31 11:58:38.603548722 (620h29m38.474631767s) [noise=true , address=77.90.31.14:4663 , piece_id=YOFV3624FNCQ66JZE2ZKHSHVPIYZPODHUTOKDFSQB5IWMBM746IA, seg_pos={0 0}, upload=543727]: Long tail cancellation due to optimal threshold Dec 31 11:58:38.603570654 (620h29m38.474653689s) [seg_pos={0 0}, upload=543727]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 11:58:38.603665946 (620h29m38.474748991s) [upload=543727]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 11:58:38.603672639 (620h29m38.474755684s) [upload=543727]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 11:58:38.889947435 (620h29m38.76103048s) [upload=543727]: Done uploading segments... Dec 31 11:58:38.889970048 (620h29m38.761053093s) [upload=543727]: Upload finished. err: <nil> Duration: 6.0040183s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........................................X...X............] duration: 13.85424198s trace: https://jaeger.dev.storj.io/trace/2e598bd65e5b1d02 FAIL (2025-12-31 12:01:33 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 13.854256159s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............................................X...X...........] duration: 25.16656067s trace: https://jaeger.dev.storj.io/trace/2e598bd65e5b1d02 FAIL (2025-12-31 12:01:19 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 25.166584365s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......X..............................................X.....] duration: 28.934986628s trace: https://jaeger.dev.storj.io/trace/3696061a2156809c FAIL (2025-12-31 12:01:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== .17:4794 , piece_id=ZCNDE7ID7HX43BDZ7KFQKZ6R5V5KUPZWCIRRUQNK65I73JMZQ2JQ, seg_pos={0 0}, upload=543695]: Uploading piece... Dec 31 11:56:12.262721887 (620h27m12.133804932s) [noise=true , address=193.149.164.143:4695 , piece_id=5CJUMEUXTJVQOCSHQKZD4ZHQBTFSMUPIQCI7QMJX2URBSUAIGIFA, seg_pos={0 0}, upload=543695]: Uploading piece... Dec 31 11:56:12.263130455 (620h27m12.1342135s) [noise=true , address=45.91.116.5:4326 , piece_id=VDLNHILWQD6ZTD7JMHQY5BDH2A4IHSJTVHJGDI4IK3OBDZSLOADA, seg_pos={0 0}, upload=543695]: Uploading piece... Dec 31 11:56:12.264175605 (620h27m12.13525865s) [noise=true , address=77.90.31.18:4812 , piece_id=KOGGG55EKOFTTT7OOZ5AULZHU2POHF5DMMSX2RM5M65FC3Y3AXRQ, seg_pos={0 0}, upload=543695]: Uploading piece... Dec 31 11:56:12.264193529 (620h27m12.135276574s) [noise=true , address=145.79.227.49:4358 , piece_id=RP2Y6L6CZD3XOL7XOU3HRDSYZZ6AENXT7LM6XVHZPIZAHSL26BPQ, seg_pos={0 0}, upload=543695]: Uploading piece... Dec 31 11:56:12.264609000 (620h27m12.135692035s) [noise=true , address=45.91.116.11:4145 , piece_id=ZX5FCIOENG6476SS6RDITOCPPCDBXAYOHAOAP4Q545FEBKSSQDDA, seg_pos={0 0}, upload=543695]: Uploading piece... Dec 31 11:56:12.264666079 (620h27m12.135749114s) [noise=true , address=64.49.13.140:4505 , piece_id=YRQ2VLRASSQVB65ONBLZXVLXY7COOSYVUUNMKJO5AOAJKFFIEW2A, seg_pos={0 0}, upload=543695]: Uploading piece... Dec 31 11:56:12.265310566 (620h27m12.136393611s) [noise=true , address=145.79.227.44:4206 , piece_id=TR747XVZACWUD6WGRSROIO5OHFLVBM3XT3XTEWA74GXNXMMYMQHQ, seg_pos={0 0}, upload=543695]: Uploading piece... Dec 31 11:56:12.265745504 (620h27m12.136828549s) [noise=true , address=145.79.227.33:4894 , piece_id=TFZVFOU3GA45OWXOWVA4R2A3W47CS6DCK5K4DAMYYMIH4J5CD23A, seg_pos={0 0}, upload=543695]: Uploading piece... Dec 31 11:56:12.266321861 (620h27m12.137404896s) [noise=true , address=45.91.116.7:4013 , piece_id=A4VDKYMR74JC325EFEP22V3TOGZRMTQB4UOE5I2Y7JKQF32AIZAA, seg_pos={0 0}, upload=543695]: Uploading piece... Dec 31 11:56:12.266868933 (620h27m12.137951978s) [noise=true , address=64.49.13.169:4707 , piece_id=ZQTZNZZQFGHXU2DIVBDUMJTYRAWAPRATLOGQ6AY4A6N45276NDXQ, seg_pos={0 0}, upload=543695]: Uploading piece... Dec 31 11:56:12.267258635 (620h27m12.13834167s) [noise=true , address=77.90.31.13:4657 , piece_id=KSNYKBDG4YOLLPTKC4OFSQR6XOBOGNA64PC3BCKQLRLYGCWFCUUQ, seg_pos={0 0}, upload=543695]: Uploading piece... Dec 31 11:56:12.267587522 (620h27m12.138670567s) [noise=true , address=145.79.227.40:4058 , piece_id=F3RLQOLC5C6JRKLBZICBMGRSJENDRTTP3PGJEFAOBXNHZP4QI3IA, seg_pos={0 0}, upload=543695]: Uploading piece... Dec 31 11:56:12.268440155 (620h27m12.13952319s) [noise=true , address=145.79.227.34:4939 , piece_id=TSCIX7EBMZ72GAOPGWHMCUW4PVEONBZ56A3ODVMIMY6LZZW2XDAQ, seg_pos={0 0}, upload=543695]: Uploading piece... Dec 31 11:56:12.269297889 (620h27m12.140380924s) [noise=true , address=145.79.227.2:4134 , piece_id=HWWXLVVLSMPGUKIP5A5T5IXAB3ZCABH7OA5DIJSNDXRQU23EIFNA, seg_pos={0 0}, upload=543695]: Uploading piece... Dec 31 11:56:13.698835153 (620h27m13.569918188s) [upload=543695]: Next returned nil segment Dec 31 11:56:13.698965801 (620h27m13.570048846s) [upload=543695]: Waiting for error group managing segments... Dec 31 11:56:14.047217086 (620h27m13.918300131s) [noise=true , address=64.49.13.140:4505 , piece_id=YRQ2VLRASSQVB65ONBLZXVLXY7COOSYVUUNMKJO5AOAJKFFIEW2A, seg_pos={0 0}, upload=543695]: Done uploading piece. err: <nil> Dec 31 11:56:14.137538488 (620h27m14.008621523s) [noise=true , address=145.79.227.2:4134 , piece_id=HWWXLVVLSMPGUKIP5A5T5IXAB3ZCABH7OA5DIJSNDXRQU23EIFNA, seg_pos={0 0}, upload=543695]: Done uploading piece. err: <nil> Dec 31 11:56:14.139917508 (620h27m14.011000543s) [noise=true , address=145.79.227.46:4619 , piece_id=BV3NLVQZM457CG2ZYJYZANIDKDVYCQWC64FZGRXFFDV6FFLC2SJA, seg_pos={0 0}, upload=543695]: Done uploading piece. err: <nil> Dec 31 11:56:14.472160101 (620h27m14.343243136s) [noise=true , address=77.90.31.17:4794 , piece_id=ZCNDE7ID7HX43BDZ7KFQKZ6R5V5KUPZWCIRRUQNK65I73JMZQ2JQ, seg_pos={0 0}, upload=543695]: Done uploading piece. err: <nil> Dec 31 11:56:14.476987874 (620h27m14.348070919s) [noise=true , address=45.91.116.5:4326 , piece_id=VDLNHILWQD6ZTD7JMHQY5BDH2A4IHSJTVHJGDI4IK3OBDZSLOADA, seg_pos={0 0}, upload=543695]: Done uploading piece. err: <nil> Dec 31 11:56:14.518004714 (620h27m14.389087749s) [noise=true , address=45.91.116.7:4013 , piece_id=A4VDKYMR74JC325EFEP22V3TOGZRMTQB4UOE5I2Y7JKQF32AIZAA, seg_pos={0 0}, upload=543695]: Done uploading piece. err: <nil> Dec 31 11:56:14.540552146 (620h27m14.411635191s) [noise=true , address=77.90.31.18:4812 , piece_id=KOGGG55EKOFTTT7OOZ5AULZHU2POHF5DMMSX2RM5M65FC3Y3AXRQ, seg_pos={0 0}, upload=543695]: Done uploading piece. err: <nil> Dec 31 11:56:14.677369210 (620h27m14.548452255s) [noise=true , address=145.79.227.40:4058 , piece_id=F3RLQOLC5C6JRKLBZICBMGRSJENDRTTP3PGJEFAOBXNHZP4QI3IA, seg_pos={0 0}, upload=543695]: Done uploading piece. err: <nil> Dec 31 11:56:15.034375707 (620h27m14.905458742s) [noise=true , address=145.79.227.44:4206 , piece_id=TR747XVZACWUD6WGRSROIO5OHFLVBM3XT3XTEWA74GXNXMMYMQHQ, seg_pos={0 0}, upload=543695]: Done uploading piece. err: <nil> Dec 31 11:56:15.095619932 (620h27m14.966702967s) [noise=true , address=145.79.227.49:4358 , piece_id=RP2Y6L6CZD3XOL7XOU3HRDSYZZ6AENXT7LM6XVHZPIZAHSL26BPQ, seg_pos={0 0}, upload=543695]: Done uploading piece. err: <nil> Dec 31 11:56:15.099851950 (620h27m14.970934995s) [noise=true , address=145.79.227.43:4164 , piece_id=QQ4D2S4XAMIDBDSZNYM2G5WT4V7DR4E5WHDSYVT4SZ4YHANAY4QA, seg_pos={0 0}, upload=543695]: Done uploading piece. err: <nil> Dec 31 11:56:15.113070049 (620h27m14.984153094s) [noise=true , address=145.79.227.33:4894 , piece_id=TFZVFOU3GA45OWXOWVA4R2A3W47CS6DCK5K4DAMYYMIH4J5CD23A, seg_pos={0 0}, upload=543695]: Done uploading piece. err: <nil> Dec 31 11:56:15.127931066 (620h27m14.999014101s) [noise=true , address=145.79.227.30:4792 , piece_id=B72NRZMLRWZQRV222OBU6EPXZYYF4K6PXT6L77XZKNVZOA25NLIQ, seg_pos={0 0}, upload=543695]: Done uploading piece. err: <nil> Dec 31 11:56:15.147826669 (620h27m15.018909714s) [noise=true , address=193.149.164.143:4695 , piece_id=5CJUMEUXTJVQOCSHQKZD4ZHQBTFSMUPIQCI7QMJX2URBSUAIGIFA, seg_pos={0 0}, upload=543695]: Done uploading piece. err: <nil> Dec 31 11:56:15.147845695 (620h27m15.01892874s) [seg_pos={0 0}, upload=543695]: Segment reached optimal threshold of 14 pieces. Dec 31 11:56:15.148175553 (620h27m15.019258598s) [noise=true , address=145.79.227.34:4939 , piece_id=TSCIX7EBMZ72GAOPGWHMCUW4PVEONBZ56A3ODVMIMY6LZZW2XDAQ, seg_pos={0 0}, upload=543695]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1h7xrPMs5kHtjm7hiB4hfDaZUHFgtJNoMGqdy3HFif2Aned8eu): protocol: expected piece hash; context canceled; EOF Dec 31 11:56:15.148194439 (620h27m15.019277484s) [noise=true , address=145.79.227.34:4939 , piece_id=TSCIX7EBMZ72GAOPGWHMCUW4PVEONBZ56A3ODVMIMY6LZZW2XDAQ, seg_pos={0 0}, upload=543695]: Long tail cancellation due to optimal threshold Dec 31 11:56:15.148491545 (620h27m15.01957458s) [noise=true , address=77.90.31.13:4657 , piece_id=KSNYKBDG4YOLLPTKC4OFSQR6XOBOGNA64PC3BCKQLRLYGCWFCUUQ, seg_pos={0 0}, upload=543695]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1LLSnWuMa6W7y2jhFUqXoSPEsTyrFsKi8FQDNXF8Tv7XHbP6yt): protocol: write tcp 188.166.225.170:49614->77.90.31.13:4657: use of closed network connection; write tcp 188.166.225.170:49614->77.90.31.13:4657: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49614->77.90.31.13:4657: use of closed network connection Dec 31 11:56:15.148498648 (620h27m15.019581693s) [noise=true , address=77.90.31.13:4657 , piece_id=KSNYKBDG4YOLLPTKC4OFSQR6XOBOGNA64PC3BCKQLRLYGCWFCUUQ, seg_pos={0 0}, upload=543695]: Long tail cancellation due to optimal threshold Dec 31 11:56:15.148641420 (620h27m15.019724465s) [noise=true , address=45.91.116.11:4145 , piece_id=ZX5FCIOENG6476SS6RDITOCPPCDBXAYOHAOAP4Q545FEBKSSQDDA, seg_pos={0 0}, upload=543695]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WKV8PJxq5Kyb3ubCbrfbyYaX6fvRkPMKW2ssUteMFNHm9fP4E): protocol: expected piece hash; context canceled; EOF Dec 31 11:56:15.148647441 (620h27m15.019730476s) [noise=true , address=45.91.116.11:4145 , piece_id=ZX5FCIOENG6476SS6RDITOCPPCDBXAYOHAOAP4Q545FEBKSSQDDA, seg_pos={0 0}, upload=543695]: Long tail cancellation due to optimal threshold Dec 31 11:56:15.148750738 (620h27m15.019833773s) [noise=true , address=64.49.13.169:4707 , piece_id=ZQTZNZZQFGHXU2DIVBDUMJTYRAWAPRATLOGQ6AY4A6N45276NDXQ, seg_pos={0 0}, upload=543695]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:45002->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:45002->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45002->64.49.13.169:4707: use of closed network connection Dec 31 11:56:15.148766589 (620h27m15.019849634s) [noise=true , address=64.49.13.169:4707 , piece_id=ZQTZNZZQFGHXU2DIVBDUMJTYRAWAPRATLOGQ6AY4A6N45276NDXQ, seg_pos={0 0}, upload=543695]: Long tail cancellation due to optimal threshold Dec 31 11:56:15.148782148 (620h27m15.019865193s) [seg_pos={0 0}, upload=543695]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 11:56:15.148845759 (620h27m15.019928804s) [upload=543695]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 11:56:15.148850428 (620h27m15.019933463s) [upload=543695]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 11:56:15.442319734 (620h27m15.313402779s) [upload=543695]: Done uploading segments... Dec 31 11:56:15.442337046 (620h27m15.313420081s) [upload=543695]: Upload finished. err: <nil> Duration: 28.935018228s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [..X....X....X................X..........X.X......X..........] duration: 3.960436743s trace: https://jaeger.dev.storj.io/trace/66084b8da1020848 FAIL (2025-12-31 12:01:10 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.960474876s core uplink us-select-1 us1-placement-20

1
...
250
251
252
253
254
...
7271