Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 23 out of 60 is failed (threshold=6) [X.......X....X...XXX......X.X...X........X.XXXXXXXXXX.X..XX.] duration: 3.840006828s trace: https://jaeger.dev.storj.io/trace/5026156ccfa61d2c FAIL (2025-12-16 12:47:44 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.840032175s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.....................XX............................X..X.....] duration: 5.868808759s trace: https://jaeger.dev.storj.io/trace/6da156ed54963bae FAIL (2025-12-16 12:47:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: CQDXY75T4WKG2LOAARKP6M4UBZ7ESH3YY2Y6TZR23OFQQBBSCK7A) context deadline exceeded; piecestore: (Node ID: 1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, Piece ID: PIM2A7BAW5JLLAUXC5KY3OIJVDUFMD2QJOMAWO2YZPDGYTV3O5UQ) context deadline exceeded ======LOGS====== Dec 16 12:43:13.728557058 (261h14m13.599640103s) [upload=229047]: Starting upload Dec 16 12:43:13.728630357 (261h14m13.599713392s) [upload=229047]: Done starting upload Dec 16 12:43:13.728720429 (261h14m13.599803464s) [upload=229047]: Uploading segments... Dec 16 12:43:13.728870865 (261h14m13.59995391s) [upload=229047]: Got next segment. Inline: false Dec 16 12:43:13.728909358 (261h14m13.599992394s) [upload=229047]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 12:43:13.728916722 (261h14m13.599999757s) [upload=229047]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 12:43:13.972957534 (261h14m13.844040579s) [seg_pos={0 0}, upload=229047]: Begin upload segment... Dec 16 12:43:13.973018189 (261h14m13.844101224s) [seg_pos={0 0}, upload=229047]: Done begin upload segment. Dec 16 12:43:13.973047195 (261h14m13.84413023s) [noise=true , address=45.91.116.10:4084 , piece_id=PIM2A7BAW5JLLAUXC5KY3OIJVDUFMD2QJOMAWO2YZPDGYTV3O5UQ, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.973077572 (261h14m13.844160607s) [noise=true , address=145.79.227.38:4979 , piece_id=ATQJMSS4OGOYOV7JHJIR5L2AXJ3JG2BDCZSH2BSMFKTCLFWBRPQQ, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.973129691 (261h14m13.844212727s) [noise=true , address=145.79.227.42:4123 , piece_id=FHBA5MWQDIO2AR2WSZKOFIOQ5CLU3KUUO6KZBI5GIYTPFSXMC6UA, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.973277132 (261h14m13.844360167s) [noise=true , address=145.79.227.15:4876 , piece_id=HTTRABRS77W63M247BPVM4355EXJ2XM5RAE4WYTU5Y57BW3KNTMA, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.974019627 (261h14m13.845102672s) [noise=true , address=145.79.227.36:4519 , piece_id=3PCQPIRPQRYNQLQ3E26DSPFEAN5OIZSPWLFNJFXG7LGSL7R5R56Q, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.974469202 (261h14m13.845552237s) [noise=true , address=145.79.227.10:4403 , piece_id=EXJQUKT3RRKVPYILF7OHM4KS65YBNJOBLZ5BDX4FQKZNPCZPLOXA, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.974574993 (261h14m13.845658038s) [noise=true , address=145.79.227.30:4762 , piece_id=43BVWF67M2D2JGE6GYLEHCNG2AWDSZAWHJD4MLUXK5SYCNY7RV2A, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.974906495 (261h14m13.84598954s) [noise=true , address=145.79.227.37:4567 , piece_id=BKXYONQBJCTHDQTOCIU4EL5DHIBFHG72ITE7DHULTGWQAOL4TGNQ, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.975608862 (261h14m13.846691907s) [noise=true , address=77.90.31.21:4877 , piece_id=HX5W43RBRV77ZVEW6PNPFJYNY42GHPP6IGM3EBYVRJOOULYTO6GQ, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.977027534 (261h14m13.848110579s) [noise=true , address=145.79.227.49:4337 , piece_id=4EBKFKOE7QQ2H3XA6ZMYVZLEIAIQI3OVIIABVNBXMNXBMJO76TVQ, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.978403523 (261h14m13.849486568s) [noise=true , address=45.91.116.8:4045 , piece_id=S5CJERIQY3GZAJXN6ZO4F7KPWG6YRIEUHZUVG32G6F2AD2IB7J6Q, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.979495323 (261h14m13.850578368s) [noise=true , address=145.79.227.23:4545 , piece_id=CMBWXQ5JQ7VMLECNCREEBSYCNEIJBLD7VURTIWE3EOAJ437255JA, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.980210935 (261h14m13.85129398s) [noise=true , address=145.79.227.45:4230 , piece_id=OXTKOHXIDLFHNWB6AGZTO46W6ISVECHVXTUMB36JC6JICYQAHPNQ, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.981410179 (261h14m13.852493214s) [noise=true , address=45.91.116.14:4190 , piece_id=CQDXY75T4WKG2LOAARKP6M4UBZ7ESH3YY2Y6TZR23OFQQBBSCK7A, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.984172609 (261h14m13.855255654s) [noise=true , address=77.90.31.10:4517 , piece_id=LCWP7QIRJEIU6ZAPN7UENDAIH2AP6PSNJBUROBVDA53TQXXUMY6A, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.984662782 (261h14m13.855745827s) [noise=true , address=145.79.227.35:4483 , piece_id=MHTKY2G6SIU2PNBESDVHCXBXHZGFR7YPDN7J3YU4CJDF5BF3VMVA, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:15.086530873 (261h14m14.957613918s) [noise=true , address=77.90.31.21:4877 , piece_id=HX5W43RBRV77ZVEW6PNPFJYNY42GHPP6IGM3EBYVRJOOULYTO6GQ, seg_pos={0 0}, upload=229047]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 16 12:43:15.669980685 (261h14m15.54106373s) [upload=229047]: Next returned nil segment Dec 16 12:43:15.669997526 (261h14m15.541080571s) [upload=229047]: Waiting for error group managing segments... Dec 16 12:43:15.903820400 (261h14m15.774903435s) [noise=true , address=77.90.31.10:4517 , piece_id=LCWP7QIRJEIU6ZAPN7UENDAIH2AP6PSNJBUROBVDA53TQXXUMY6A, seg_pos={0 0}, upload=229047]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: storage node overloaded, request limit: 2000 Dec 16 12:43:15.951902859 (261h14m15.822985904s) [noise=true , address=145.79.227.10:4403 , piece_id=EXJQUKT3RRKVPYILF7OHM4KS65YBNJOBLZ5BDX4FQKZNPCZPLOXA, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:15.973291127 (261h14m15.844374162s) [noise=true , address=145.79.227.42:4123 , piece_id=FHBA5MWQDIO2AR2WSZKOFIOQ5CLU3KUUO6KZBI5GIYTPFSXMC6UA, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:15.973360790 (261h14m15.844443835s) [noise=true , address=145.79.227.23:4545 , piece_id=CMBWXQ5JQ7VMLECNCREEBSYCNEIJBLD7VURTIWE3EOAJ437255JA, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:15.982520812 (261h14m15.853603857s) [noise=true , address=145.79.227.38:4979 , piece_id=ATQJMSS4OGOYOV7JHJIR5L2AXJ3JG2BDCZSH2BSMFKTCLFWBRPQQ, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:15.984472408 (261h14m15.855555443s) [noise=true , address=145.79.227.49:4337 , piece_id=4EBKFKOE7QQ2H3XA6ZMYVZLEIAIQI3OVIIABVNBXMNXBMJO76TVQ, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:15.986032358 (261h14m15.857115403s) [noise=true , address=145.79.227.15:4876 , piece_id=HTTRABRS77W63M247BPVM4355EXJ2XM5RAE4WYTU5Y57BW3KNTMA, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:15.986742190 (261h14m15.857825225s) [noise=true , address=145.79.227.35:4483 , piece_id=MHTKY2G6SIU2PNBESDVHCXBXHZGFR7YPDN7J3YU4CJDF5BF3VMVA, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:15.996623816 (261h14m15.867706851s) [noise=true , address=145.79.227.45:4230 , piece_id=OXTKOHXIDLFHNWB6AGZTO46W6ISVECHVXTUMB36JC6JICYQAHPNQ, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:16.051285142 (261h14m15.922368178s) [noise=true , address=145.79.227.36:4519 , piece_id=3PCQPIRPQRYNQLQ3E26DSPFEAN5OIZSPWLFNJFXG7LGSL7R5R56Q, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:16.055795661 (261h14m15.926878696s) [noise=true , address=145.79.227.37:4567 , piece_id=BKXYONQBJCTHDQTOCIU4EL5DHIBFHG72ITE7DHULTGWQAOL4TGNQ, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:16.292043703 (261h14m16.163126758s) [noise=true , address=45.91.116.14:4190 , piece_id=CQDXY75T4WKG2LOAARKP6M4UBZ7ESH3YY2Y6TZR23OFQQBBSCK7A, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:16.323293028 (261h14m16.194376073s) [noise=true , address=45.91.116.8:4045 , piece_id=S5CJERIQY3GZAJXN6ZO4F7KPWG6YRIEUHZUVG32G6F2AD2IB7J6Q, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:17.080613814 (261h14m16.951696859s) [noise=true , address=45.91.116.10:4084 , piece_id=PIM2A7BAW5JLLAUXC5KY3OIJVDUFMD2QJOMAWO2YZPDGYTV3O5UQ, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:17.080636187 (261h14m16.951719222s) [seg_pos={0 0}, upload=229047]: Segment reached optimal threshold of 13 pieces. Dec 16 12:43:17.080908635 (261h14m16.95199169s) [noise=true , address=145.79.227.30:4762 , piece_id=43BVWF67M2D2JGE6GYLEHCNG2AWDSZAWHJD4MLUXK5SYCNY7RV2A, seg_pos={0 0}, upload=229047]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Dec 16 12:43:17.080919286 (261h14m16.952002321s) [noise=true , address=145.79.227.30:4762 , piece_id=43BVWF67M2D2JGE6GYLEHCNG2AWDSZAWHJD4MLUXK5SYCNY7RV2A, seg_pos={0 0}, upload=229047]: Long tail cancellation due to optimal threshold Dec 16 12:43:17.080942920 (261h14m16.952025955s) [seg_pos={0 0}, upload=229047]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 12:43:17.081036538 (261h14m16.952119583s) [upload=229047]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 12:43:17.081043632 (261h14m16.952126677s) [upload=229047]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 12:43:17.371280229 (261h14m17.242363264s) [upload=229047]: Done uploading segments... Dec 16 12:43:17.371312330 (261h14m17.242395375s) [upload=229047]: Upload finished. err: <nil> Duration: 5.868849096s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 24 out of 60 is failed (threshold=6) [XX.......X....X...XXX......X.X...X........X.XXXXXXXXXX.X..XX] duration: 5.001669781s trace: https://jaeger.dev.storj.io/trace/5026156ccfa61d2c FAIL (2025-12-16 12:47:02 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001695861s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [......................XX............................X..X....] duration: 5.962684275s trace: https://jaeger.dev.storj.io/trace/6da156ed54963bae FAIL (2025-12-16 12:46:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: CQDXY75T4WKG2LOAARKP6M4UBZ7ESH3YY2Y6TZR23OFQQBBSCK7A) context deadline exceeded; piecestore: (Node ID: 1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, Piece ID: PIM2A7BAW5JLLAUXC5KY3OIJVDUFMD2QJOMAWO2YZPDGYTV3O5UQ) context deadline exceeded ======LOGS====== Dec 16 12:43:13.728557058 (261h14m13.599640103s) [upload=229047]: Starting upload Dec 16 12:43:13.728630357 (261h14m13.599713392s) [upload=229047]: Done starting upload Dec 16 12:43:13.728720429 (261h14m13.599803464s) [upload=229047]: Uploading segments... Dec 16 12:43:13.728870865 (261h14m13.59995391s) [upload=229047]: Got next segment. Inline: false Dec 16 12:43:13.728909358 (261h14m13.599992394s) [upload=229047]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 12:43:13.728916722 (261h14m13.599999757s) [upload=229047]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 12:43:13.972957534 (261h14m13.844040579s) [seg_pos={0 0}, upload=229047]: Begin upload segment... Dec 16 12:43:13.973018189 (261h14m13.844101224s) [seg_pos={0 0}, upload=229047]: Done begin upload segment. Dec 16 12:43:13.973047195 (261h14m13.84413023s) [noise=true , address=45.91.116.10:4084 , piece_id=PIM2A7BAW5JLLAUXC5KY3OIJVDUFMD2QJOMAWO2YZPDGYTV3O5UQ, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.973077572 (261h14m13.844160607s) [noise=true , address=145.79.227.38:4979 , piece_id=ATQJMSS4OGOYOV7JHJIR5L2AXJ3JG2BDCZSH2BSMFKTCLFWBRPQQ, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.973129691 (261h14m13.844212727s) [noise=true , address=145.79.227.42:4123 , piece_id=FHBA5MWQDIO2AR2WSZKOFIOQ5CLU3KUUO6KZBI5GIYTPFSXMC6UA, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.973277132 (261h14m13.844360167s) [noise=true , address=145.79.227.15:4876 , piece_id=HTTRABRS77W63M247BPVM4355EXJ2XM5RAE4WYTU5Y57BW3KNTMA, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.974019627 (261h14m13.845102672s) [noise=true , address=145.79.227.36:4519 , piece_id=3PCQPIRPQRYNQLQ3E26DSPFEAN5OIZSPWLFNJFXG7LGSL7R5R56Q, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.974469202 (261h14m13.845552237s) [noise=true , address=145.79.227.10:4403 , piece_id=EXJQUKT3RRKVPYILF7OHM4KS65YBNJOBLZ5BDX4FQKZNPCZPLOXA, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.974574993 (261h14m13.845658038s) [noise=true , address=145.79.227.30:4762 , piece_id=43BVWF67M2D2JGE6GYLEHCNG2AWDSZAWHJD4MLUXK5SYCNY7RV2A, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.974906495 (261h14m13.84598954s) [noise=true , address=145.79.227.37:4567 , piece_id=BKXYONQBJCTHDQTOCIU4EL5DHIBFHG72ITE7DHULTGWQAOL4TGNQ, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.975608862 (261h14m13.846691907s) [noise=true , address=77.90.31.21:4877 , piece_id=HX5W43RBRV77ZVEW6PNPFJYNY42GHPP6IGM3EBYVRJOOULYTO6GQ, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.977027534 (261h14m13.848110579s) [noise=true , address=145.79.227.49:4337 , piece_id=4EBKFKOE7QQ2H3XA6ZMYVZLEIAIQI3OVIIABVNBXMNXBMJO76TVQ, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.978403523 (261h14m13.849486568s) [noise=true , address=45.91.116.8:4045 , piece_id=S5CJERIQY3GZAJXN6ZO4F7KPWG6YRIEUHZUVG32G6F2AD2IB7J6Q, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.979495323 (261h14m13.850578368s) [noise=true , address=145.79.227.23:4545 , piece_id=CMBWXQ5JQ7VMLECNCREEBSYCNEIJBLD7VURTIWE3EOAJ437255JA, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.980210935 (261h14m13.85129398s) [noise=true , address=145.79.227.45:4230 , piece_id=OXTKOHXIDLFHNWB6AGZTO46W6ISVECHVXTUMB36JC6JICYQAHPNQ, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.981410179 (261h14m13.852493214s) [noise=true , address=45.91.116.14:4190 , piece_id=CQDXY75T4WKG2LOAARKP6M4UBZ7ESH3YY2Y6TZR23OFQQBBSCK7A, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.984172609 (261h14m13.855255654s) [noise=true , address=77.90.31.10:4517 , piece_id=LCWP7QIRJEIU6ZAPN7UENDAIH2AP6PSNJBUROBVDA53TQXXUMY6A, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.984662782 (261h14m13.855745827s) [noise=true , address=145.79.227.35:4483 , piece_id=MHTKY2G6SIU2PNBESDVHCXBXHZGFR7YPDN7J3YU4CJDF5BF3VMVA, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:15.086530873 (261h14m14.957613918s) [noise=true , address=77.90.31.21:4877 , piece_id=HX5W43RBRV77ZVEW6PNPFJYNY42GHPP6IGM3EBYVRJOOULYTO6GQ, seg_pos={0 0}, upload=229047]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 16 12:43:15.669980685 (261h14m15.54106373s) [upload=229047]: Next returned nil segment Dec 16 12:43:15.669997526 (261h14m15.541080571s) [upload=229047]: Waiting for error group managing segments... Dec 16 12:43:15.903820400 (261h14m15.774903435s) [noise=true , address=77.90.31.10:4517 , piece_id=LCWP7QIRJEIU6ZAPN7UENDAIH2AP6PSNJBUROBVDA53TQXXUMY6A, seg_pos={0 0}, upload=229047]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: storage node overloaded, request limit: 2000 Dec 16 12:43:15.951902859 (261h14m15.822985904s) [noise=true , address=145.79.227.10:4403 , piece_id=EXJQUKT3RRKVPYILF7OHM4KS65YBNJOBLZ5BDX4FQKZNPCZPLOXA, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:15.973291127 (261h14m15.844374162s) [noise=true , address=145.79.227.42:4123 , piece_id=FHBA5MWQDIO2AR2WSZKOFIOQ5CLU3KUUO6KZBI5GIYTPFSXMC6UA, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:15.973360790 (261h14m15.844443835s) [noise=true , address=145.79.227.23:4545 , piece_id=CMBWXQ5JQ7VMLECNCREEBSYCNEIJBLD7VURTIWE3EOAJ437255JA, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:15.982520812 (261h14m15.853603857s) [noise=true , address=145.79.227.38:4979 , piece_id=ATQJMSS4OGOYOV7JHJIR5L2AXJ3JG2BDCZSH2BSMFKTCLFWBRPQQ, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:15.984472408 (261h14m15.855555443s) [noise=true , address=145.79.227.49:4337 , piece_id=4EBKFKOE7QQ2H3XA6ZMYVZLEIAIQI3OVIIABVNBXMNXBMJO76TVQ, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:15.986032358 (261h14m15.857115403s) [noise=true , address=145.79.227.15:4876 , piece_id=HTTRABRS77W63M247BPVM4355EXJ2XM5RAE4WYTU5Y57BW3KNTMA, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:15.986742190 (261h14m15.857825225s) [noise=true , address=145.79.227.35:4483 , piece_id=MHTKY2G6SIU2PNBESDVHCXBXHZGFR7YPDN7J3YU4CJDF5BF3VMVA, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:15.996623816 (261h14m15.867706851s) [noise=true , address=145.79.227.45:4230 , piece_id=OXTKOHXIDLFHNWB6AGZTO46W6ISVECHVXTUMB36JC6JICYQAHPNQ, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:16.051285142 (261h14m15.922368178s) [noise=true , address=145.79.227.36:4519 , piece_id=3PCQPIRPQRYNQLQ3E26DSPFEAN5OIZSPWLFNJFXG7LGSL7R5R56Q, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:16.055795661 (261h14m15.926878696s) [noise=true , address=145.79.227.37:4567 , piece_id=BKXYONQBJCTHDQTOCIU4EL5DHIBFHG72ITE7DHULTGWQAOL4TGNQ, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:16.292043703 (261h14m16.163126758s) [noise=true , address=45.91.116.14:4190 , piece_id=CQDXY75T4WKG2LOAARKP6M4UBZ7ESH3YY2Y6TZR23OFQQBBSCK7A, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:16.323293028 (261h14m16.194376073s) [noise=true , address=45.91.116.8:4045 , piece_id=S5CJERIQY3GZAJXN6ZO4F7KPWG6YRIEUHZUVG32G6F2AD2IB7J6Q, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:17.080613814 (261h14m16.951696859s) [noise=true , address=45.91.116.10:4084 , piece_id=PIM2A7BAW5JLLAUXC5KY3OIJVDUFMD2QJOMAWO2YZPDGYTV3O5UQ, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:17.080636187 (261h14m16.951719222s) [seg_pos={0 0}, upload=229047]: Segment reached optimal threshold of 13 pieces. Dec 16 12:43:17.080908635 (261h14m16.95199169s) [noise=true , address=145.79.227.30:4762 , piece_id=43BVWF67M2D2JGE6GYLEHCNG2AWDSZAWHJD4MLUXK5SYCNY7RV2A, seg_pos={0 0}, upload=229047]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Dec 16 12:43:17.080919286 (261h14m16.952002321s) [noise=true , address=145.79.227.30:4762 , piece_id=43BVWF67M2D2JGE6GYLEHCNG2AWDSZAWHJD4MLUXK5SYCNY7RV2A, seg_pos={0 0}, upload=229047]: Long tail cancellation due to optimal threshold Dec 16 12:43:17.080942920 (261h14m16.952025955s) [seg_pos={0 0}, upload=229047]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 12:43:17.081036538 (261h14m16.952119583s) [upload=229047]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 12:43:17.081043632 (261h14m16.952126677s) [upload=229047]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 12:43:17.371280229 (261h14m17.242363264s) [upload=229047]: Done uploading segments... Dec 16 12:43:17.371312330 (261h14m17.242395375s) [upload=229047]: Upload finished. err: <nil> Duration: 5.962711489s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 23 out of 60 is failed (threshold=6) [.XX.......X....X...XXX......X.X...X........X.XXXXXXXXXX.X..X] duration: 5.00136522s trace: https://jaeger.dev.storj.io/trace/15deb913f40e1127 FAIL (2025-12-16 12:46:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001390648s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 23 out of 60 is failed (threshold=6) [X.XX.......X....X...XXX......X.X...X........X.XXXXXXXXXX.X..] duration: 4.323236508s trace: https://jaeger.dev.storj.io/trace/59e178e7f0b1e61d FAIL (2025-12-16 12:45:34 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.323260654s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.......................XX............................X..X...] duration: 5.932717355s trace: https://jaeger.dev.storj.io/trace/6da156ed54963bae FAIL (2025-12-16 12:45:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: CQDXY75T4WKG2LOAARKP6M4UBZ7ESH3YY2Y6TZR23OFQQBBSCK7A) context deadline exceeded; piecestore: (Node ID: 1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, Piece ID: PIM2A7BAW5JLLAUXC5KY3OIJVDUFMD2QJOMAWO2YZPDGYTV3O5UQ) context deadline exceeded ======LOGS====== Dec 16 12:43:13.728557058 (261h14m13.599640103s) [upload=229047]: Starting upload Dec 16 12:43:13.728630357 (261h14m13.599713392s) [upload=229047]: Done starting upload Dec 16 12:43:13.728720429 (261h14m13.599803464s) [upload=229047]: Uploading segments... Dec 16 12:43:13.728870865 (261h14m13.59995391s) [upload=229047]: Got next segment. Inline: false Dec 16 12:43:13.728909358 (261h14m13.599992394s) [upload=229047]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 12:43:13.728916722 (261h14m13.599999757s) [upload=229047]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 12:43:13.972957534 (261h14m13.844040579s) [seg_pos={0 0}, upload=229047]: Begin upload segment... Dec 16 12:43:13.973018189 (261h14m13.844101224s) [seg_pos={0 0}, upload=229047]: Done begin upload segment. Dec 16 12:43:13.973047195 (261h14m13.84413023s) [noise=true , address=45.91.116.10:4084 , piece_id=PIM2A7BAW5JLLAUXC5KY3OIJVDUFMD2QJOMAWO2YZPDGYTV3O5UQ, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.973077572 (261h14m13.844160607s) [noise=true , address=145.79.227.38:4979 , piece_id=ATQJMSS4OGOYOV7JHJIR5L2AXJ3JG2BDCZSH2BSMFKTCLFWBRPQQ, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.973129691 (261h14m13.844212727s) [noise=true , address=145.79.227.42:4123 , piece_id=FHBA5MWQDIO2AR2WSZKOFIOQ5CLU3KUUO6KZBI5GIYTPFSXMC6UA, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.973277132 (261h14m13.844360167s) [noise=true , address=145.79.227.15:4876 , piece_id=HTTRABRS77W63M247BPVM4355EXJ2XM5RAE4WYTU5Y57BW3KNTMA, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.974019627 (261h14m13.845102672s) [noise=true , address=145.79.227.36:4519 , piece_id=3PCQPIRPQRYNQLQ3E26DSPFEAN5OIZSPWLFNJFXG7LGSL7R5R56Q, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.974469202 (261h14m13.845552237s) [noise=true , address=145.79.227.10:4403 , piece_id=EXJQUKT3RRKVPYILF7OHM4KS65YBNJOBLZ5BDX4FQKZNPCZPLOXA, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.974574993 (261h14m13.845658038s) [noise=true , address=145.79.227.30:4762 , piece_id=43BVWF67M2D2JGE6GYLEHCNG2AWDSZAWHJD4MLUXK5SYCNY7RV2A, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.974906495 (261h14m13.84598954s) [noise=true , address=145.79.227.37:4567 , piece_id=BKXYONQBJCTHDQTOCIU4EL5DHIBFHG72ITE7DHULTGWQAOL4TGNQ, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.975608862 (261h14m13.846691907s) [noise=true , address=77.90.31.21:4877 , piece_id=HX5W43RBRV77ZVEW6PNPFJYNY42GHPP6IGM3EBYVRJOOULYTO6GQ, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.977027534 (261h14m13.848110579s) [noise=true , address=145.79.227.49:4337 , piece_id=4EBKFKOE7QQ2H3XA6ZMYVZLEIAIQI3OVIIABVNBXMNXBMJO76TVQ, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.978403523 (261h14m13.849486568s) [noise=true , address=45.91.116.8:4045 , piece_id=S5CJERIQY3GZAJXN6ZO4F7KPWG6YRIEUHZUVG32G6F2AD2IB7J6Q, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.979495323 (261h14m13.850578368s) [noise=true , address=145.79.227.23:4545 , piece_id=CMBWXQ5JQ7VMLECNCREEBSYCNEIJBLD7VURTIWE3EOAJ437255JA, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.980210935 (261h14m13.85129398s) [noise=true , address=145.79.227.45:4230 , piece_id=OXTKOHXIDLFHNWB6AGZTO46W6ISVECHVXTUMB36JC6JICYQAHPNQ, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.981410179 (261h14m13.852493214s) [noise=true , address=45.91.116.14:4190 , piece_id=CQDXY75T4WKG2LOAARKP6M4UBZ7ESH3YY2Y6TZR23OFQQBBSCK7A, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.984172609 (261h14m13.855255654s) [noise=true , address=77.90.31.10:4517 , piece_id=LCWP7QIRJEIU6ZAPN7UENDAIH2AP6PSNJBUROBVDA53TQXXUMY6A, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.984662782 (261h14m13.855745827s) [noise=true , address=145.79.227.35:4483 , piece_id=MHTKY2G6SIU2PNBESDVHCXBXHZGFR7YPDN7J3YU4CJDF5BF3VMVA, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:15.086530873 (261h14m14.957613918s) [noise=true , address=77.90.31.21:4877 , piece_id=HX5W43RBRV77ZVEW6PNPFJYNY42GHPP6IGM3EBYVRJOOULYTO6GQ, seg_pos={0 0}, upload=229047]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 16 12:43:15.669980685 (261h14m15.54106373s) [upload=229047]: Next returned nil segment Dec 16 12:43:15.669997526 (261h14m15.541080571s) [upload=229047]: Waiting for error group managing segments... Dec 16 12:43:15.903820400 (261h14m15.774903435s) [noise=true , address=77.90.31.10:4517 , piece_id=LCWP7QIRJEIU6ZAPN7UENDAIH2AP6PSNJBUROBVDA53TQXXUMY6A, seg_pos={0 0}, upload=229047]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: storage node overloaded, request limit: 2000 Dec 16 12:43:15.951902859 (261h14m15.822985904s) [noise=true , address=145.79.227.10:4403 , piece_id=EXJQUKT3RRKVPYILF7OHM4KS65YBNJOBLZ5BDX4FQKZNPCZPLOXA, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:15.973291127 (261h14m15.844374162s) [noise=true , address=145.79.227.42:4123 , piece_id=FHBA5MWQDIO2AR2WSZKOFIOQ5CLU3KUUO6KZBI5GIYTPFSXMC6UA, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:15.973360790 (261h14m15.844443835s) [noise=true , address=145.79.227.23:4545 , piece_id=CMBWXQ5JQ7VMLECNCREEBSYCNEIJBLD7VURTIWE3EOAJ437255JA, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:15.982520812 (261h14m15.853603857s) [noise=true , address=145.79.227.38:4979 , piece_id=ATQJMSS4OGOYOV7JHJIR5L2AXJ3JG2BDCZSH2BSMFKTCLFWBRPQQ, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:15.984472408 (261h14m15.855555443s) [noise=true , address=145.79.227.49:4337 , piece_id=4EBKFKOE7QQ2H3XA6ZMYVZLEIAIQI3OVIIABVNBXMNXBMJO76TVQ, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:15.986032358 (261h14m15.857115403s) [noise=true , address=145.79.227.15:4876 , piece_id=HTTRABRS77W63M247BPVM4355EXJ2XM5RAE4WYTU5Y57BW3KNTMA, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:15.986742190 (261h14m15.857825225s) [noise=true , address=145.79.227.35:4483 , piece_id=MHTKY2G6SIU2PNBESDVHCXBXHZGFR7YPDN7J3YU4CJDF5BF3VMVA, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:15.996623816 (261h14m15.867706851s) [noise=true , address=145.79.227.45:4230 , piece_id=OXTKOHXIDLFHNWB6AGZTO46W6ISVECHVXTUMB36JC6JICYQAHPNQ, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:16.051285142 (261h14m15.922368178s) [noise=true , address=145.79.227.36:4519 , piece_id=3PCQPIRPQRYNQLQ3E26DSPFEAN5OIZSPWLFNJFXG7LGSL7R5R56Q, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:16.055795661 (261h14m15.926878696s) [noise=true , address=145.79.227.37:4567 , piece_id=BKXYONQBJCTHDQTOCIU4EL5DHIBFHG72ITE7DHULTGWQAOL4TGNQ, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:16.292043703 (261h14m16.163126758s) [noise=true , address=45.91.116.14:4190 , piece_id=CQDXY75T4WKG2LOAARKP6M4UBZ7ESH3YY2Y6TZR23OFQQBBSCK7A, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:16.323293028 (261h14m16.194376073s) [noise=true , address=45.91.116.8:4045 , piece_id=S5CJERIQY3GZAJXN6ZO4F7KPWG6YRIEUHZUVG32G6F2AD2IB7J6Q, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:17.080613814 (261h14m16.951696859s) [noise=true , address=45.91.116.10:4084 , piece_id=PIM2A7BAW5JLLAUXC5KY3OIJVDUFMD2QJOMAWO2YZPDGYTV3O5UQ, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:17.080636187 (261h14m16.951719222s) [seg_pos={0 0}, upload=229047]: Segment reached optimal threshold of 13 pieces. Dec 16 12:43:17.080908635 (261h14m16.95199169s) [noise=true , address=145.79.227.30:4762 , piece_id=43BVWF67M2D2JGE6GYLEHCNG2AWDSZAWHJD4MLUXK5SYCNY7RV2A, seg_pos={0 0}, upload=229047]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Dec 16 12:43:17.080919286 (261h14m16.952002321s) [noise=true , address=145.79.227.30:4762 , piece_id=43BVWF67M2D2JGE6GYLEHCNG2AWDSZAWHJD4MLUXK5SYCNY7RV2A, seg_pos={0 0}, upload=229047]: Long tail cancellation due to optimal threshold Dec 16 12:43:17.080942920 (261h14m16.952025955s) [seg_pos={0 0}, upload=229047]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 12:43:17.081036538 (261h14m16.952119583s) [upload=229047]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 12:43:17.081043632 (261h14m16.952126677s) [upload=229047]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 12:43:17.371280229 (261h14m17.242363264s) [upload=229047]: Done uploading segments... Dec 16 12:43:17.371312330 (261h14m17.242395375s) [upload=229047]: Upload finished. err: <nil> Duration: 5.932750498s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 23 out of 60 is failed (threshold=6) [.X.XX.......X....X...XXX......X.X...X........X.XXXXXXXXXX.X.] duration: 3.870233604s trace: https://jaeger.dev.storj.io/trace/59e178e7f0b1e61d FAIL (2025-12-16 12:44:48 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.870260517s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 23 out of 60 is failed (threshold=6) [..X.XX.......X....X...XXX......X.X...X........X.XXXXXXXXXX.X] duration: 5.001168751s trace: https://jaeger.dev.storj.io/trace/59e178e7f0b1e61d FAIL (2025-12-16 12:44:28 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001190732s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [........................XX............................X..X..] duration: 5.795054041s trace: https://jaeger.dev.storj.io/trace/6da156ed54963bae FAIL (2025-12-16 12:44:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: CQDXY75T4WKG2LOAARKP6M4UBZ7ESH3YY2Y6TZR23OFQQBBSCK7A) context deadline exceeded; piecestore: (Node ID: 1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, Piece ID: PIM2A7BAW5JLLAUXC5KY3OIJVDUFMD2QJOMAWO2YZPDGYTV3O5UQ) context deadline exceeded ======LOGS====== Dec 16 12:43:13.728557058 (261h14m13.599640103s) [upload=229047]: Starting upload Dec 16 12:43:13.728630357 (261h14m13.599713392s) [upload=229047]: Done starting upload Dec 16 12:43:13.728720429 (261h14m13.599803464s) [upload=229047]: Uploading segments... Dec 16 12:43:13.728870865 (261h14m13.59995391s) [upload=229047]: Got next segment. Inline: false Dec 16 12:43:13.728909358 (261h14m13.599992394s) [upload=229047]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 12:43:13.728916722 (261h14m13.599999757s) [upload=229047]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 12:43:13.972957534 (261h14m13.844040579s) [seg_pos={0 0}, upload=229047]: Begin upload segment... Dec 16 12:43:13.973018189 (261h14m13.844101224s) [seg_pos={0 0}, upload=229047]: Done begin upload segment. Dec 16 12:43:13.973047195 (261h14m13.84413023s) [noise=true , address=45.91.116.10:4084 , piece_id=PIM2A7BAW5JLLAUXC5KY3OIJVDUFMD2QJOMAWO2YZPDGYTV3O5UQ, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.973077572 (261h14m13.844160607s) [noise=true , address=145.79.227.38:4979 , piece_id=ATQJMSS4OGOYOV7JHJIR5L2AXJ3JG2BDCZSH2BSMFKTCLFWBRPQQ, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.973129691 (261h14m13.844212727s) [noise=true , address=145.79.227.42:4123 , piece_id=FHBA5MWQDIO2AR2WSZKOFIOQ5CLU3KUUO6KZBI5GIYTPFSXMC6UA, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.973277132 (261h14m13.844360167s) [noise=true , address=145.79.227.15:4876 , piece_id=HTTRABRS77W63M247BPVM4355EXJ2XM5RAE4WYTU5Y57BW3KNTMA, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.974019627 (261h14m13.845102672s) [noise=true , address=145.79.227.36:4519 , piece_id=3PCQPIRPQRYNQLQ3E26DSPFEAN5OIZSPWLFNJFXG7LGSL7R5R56Q, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.974469202 (261h14m13.845552237s) [noise=true , address=145.79.227.10:4403 , piece_id=EXJQUKT3RRKVPYILF7OHM4KS65YBNJOBLZ5BDX4FQKZNPCZPLOXA, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.974574993 (261h14m13.845658038s) [noise=true , address=145.79.227.30:4762 , piece_id=43BVWF67M2D2JGE6GYLEHCNG2AWDSZAWHJD4MLUXK5SYCNY7RV2A, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.974906495 (261h14m13.84598954s) [noise=true , address=145.79.227.37:4567 , piece_id=BKXYONQBJCTHDQTOCIU4EL5DHIBFHG72ITE7DHULTGWQAOL4TGNQ, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.975608862 (261h14m13.846691907s) [noise=true , address=77.90.31.21:4877 , piece_id=HX5W43RBRV77ZVEW6PNPFJYNY42GHPP6IGM3EBYVRJOOULYTO6GQ, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.977027534 (261h14m13.848110579s) [noise=true , address=145.79.227.49:4337 , piece_id=4EBKFKOE7QQ2H3XA6ZMYVZLEIAIQI3OVIIABVNBXMNXBMJO76TVQ, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.978403523 (261h14m13.849486568s) [noise=true , address=45.91.116.8:4045 , piece_id=S5CJERIQY3GZAJXN6ZO4F7KPWG6YRIEUHZUVG32G6F2AD2IB7J6Q, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.979495323 (261h14m13.850578368s) [noise=true , address=145.79.227.23:4545 , piece_id=CMBWXQ5JQ7VMLECNCREEBSYCNEIJBLD7VURTIWE3EOAJ437255JA, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.980210935 (261h14m13.85129398s) [noise=true , address=145.79.227.45:4230 , piece_id=OXTKOHXIDLFHNWB6AGZTO46W6ISVECHVXTUMB36JC6JICYQAHPNQ, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.981410179 (261h14m13.852493214s) [noise=true , address=45.91.116.14:4190 , piece_id=CQDXY75T4WKG2LOAARKP6M4UBZ7ESH3YY2Y6TZR23OFQQBBSCK7A, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.984172609 (261h14m13.855255654s) [noise=true , address=77.90.31.10:4517 , piece_id=LCWP7QIRJEIU6ZAPN7UENDAIH2AP6PSNJBUROBVDA53TQXXUMY6A, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.984662782 (261h14m13.855745827s) [noise=true , address=145.79.227.35:4483 , piece_id=MHTKY2G6SIU2PNBESDVHCXBXHZGFR7YPDN7J3YU4CJDF5BF3VMVA, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:15.086530873 (261h14m14.957613918s) [noise=true , address=77.90.31.21:4877 , piece_id=HX5W43RBRV77ZVEW6PNPFJYNY42GHPP6IGM3EBYVRJOOULYTO6GQ, seg_pos={0 0}, upload=229047]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 16 12:43:15.669980685 (261h14m15.54106373s) [upload=229047]: Next returned nil segment Dec 16 12:43:15.669997526 (261h14m15.541080571s) [upload=229047]: Waiting for error group managing segments... Dec 16 12:43:15.903820400 (261h14m15.774903435s) [noise=true , address=77.90.31.10:4517 , piece_id=LCWP7QIRJEIU6ZAPN7UENDAIH2AP6PSNJBUROBVDA53TQXXUMY6A, seg_pos={0 0}, upload=229047]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: storage node overloaded, request limit: 2000 Dec 16 12:43:15.951902859 (261h14m15.822985904s) [noise=true , address=145.79.227.10:4403 , piece_id=EXJQUKT3RRKVPYILF7OHM4KS65YBNJOBLZ5BDX4FQKZNPCZPLOXA, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:15.973291127 (261h14m15.844374162s) [noise=true , address=145.79.227.42:4123 , piece_id=FHBA5MWQDIO2AR2WSZKOFIOQ5CLU3KUUO6KZBI5GIYTPFSXMC6UA, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:15.973360790 (261h14m15.844443835s) [noise=true , address=145.79.227.23:4545 , piece_id=CMBWXQ5JQ7VMLECNCREEBSYCNEIJBLD7VURTIWE3EOAJ437255JA, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:15.982520812 (261h14m15.853603857s) [noise=true , address=145.79.227.38:4979 , piece_id=ATQJMSS4OGOYOV7JHJIR5L2AXJ3JG2BDCZSH2BSMFKTCLFWBRPQQ, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:15.984472408 (261h14m15.855555443s) [noise=true , address=145.79.227.49:4337 , piece_id=4EBKFKOE7QQ2H3XA6ZMYVZLEIAIQI3OVIIABVNBXMNXBMJO76TVQ, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:15.986032358 (261h14m15.857115403s) [noise=true , address=145.79.227.15:4876 , piece_id=HTTRABRS77W63M247BPVM4355EXJ2XM5RAE4WYTU5Y57BW3KNTMA, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:15.986742190 (261h14m15.857825225s) [noise=true , address=145.79.227.35:4483 , piece_id=MHTKY2G6SIU2PNBESDVHCXBXHZGFR7YPDN7J3YU4CJDF5BF3VMVA, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:15.996623816 (261h14m15.867706851s) [noise=true , address=145.79.227.45:4230 , piece_id=OXTKOHXIDLFHNWB6AGZTO46W6ISVECHVXTUMB36JC6JICYQAHPNQ, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:16.051285142 (261h14m15.922368178s) [noise=true , address=145.79.227.36:4519 , piece_id=3PCQPIRPQRYNQLQ3E26DSPFEAN5OIZSPWLFNJFXG7LGSL7R5R56Q, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:16.055795661 (261h14m15.926878696s) [noise=true , address=145.79.227.37:4567 , piece_id=BKXYONQBJCTHDQTOCIU4EL5DHIBFHG72ITE7DHULTGWQAOL4TGNQ, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:16.292043703 (261h14m16.163126758s) [noise=true , address=45.91.116.14:4190 , piece_id=CQDXY75T4WKG2LOAARKP6M4UBZ7ESH3YY2Y6TZR23OFQQBBSCK7A, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:16.323293028 (261h14m16.194376073s) [noise=true , address=45.91.116.8:4045 , piece_id=S5CJERIQY3GZAJXN6ZO4F7KPWG6YRIEUHZUVG32G6F2AD2IB7J6Q, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:17.080613814 (261h14m16.951696859s) [noise=true , address=45.91.116.10:4084 , piece_id=PIM2A7BAW5JLLAUXC5KY3OIJVDUFMD2QJOMAWO2YZPDGYTV3O5UQ, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:17.080636187 (261h14m16.951719222s) [seg_pos={0 0}, upload=229047]: Segment reached optimal threshold of 13 pieces. Dec 16 12:43:17.080908635 (261h14m16.95199169s) [noise=true , address=145.79.227.30:4762 , piece_id=43BVWF67M2D2JGE6GYLEHCNG2AWDSZAWHJD4MLUXK5SYCNY7RV2A, seg_pos={0 0}, upload=229047]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Dec 16 12:43:17.080919286 (261h14m16.952002321s) [noise=true , address=145.79.227.30:4762 , piece_id=43BVWF67M2D2JGE6GYLEHCNG2AWDSZAWHJD4MLUXK5SYCNY7RV2A, seg_pos={0 0}, upload=229047]: Long tail cancellation due to optimal threshold Dec 16 12:43:17.080942920 (261h14m16.952025955s) [seg_pos={0 0}, upload=229047]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 12:43:17.081036538 (261h14m16.952119583s) [upload=229047]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 12:43:17.081043632 (261h14m16.952126677s) [upload=229047]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 12:43:17.371280229 (261h14m17.242363264s) [upload=229047]: Done uploading segments... Dec 16 12:43:17.371312330 (261h14m17.242395375s) [upload=229047]: Upload finished. err: <nil> Duration: 5.795082394s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.........................XX............................X..X.] duration: 6.33917626s trace: https://jaeger.dev.storj.io/trace/6da156ed54963bae FAIL (2025-12-16 12:43:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: CQDXY75T4WKG2LOAARKP6M4UBZ7ESH3YY2Y6TZR23OFQQBBSCK7A) context deadline exceeded; piecestore: (Node ID: 1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, Piece ID: PIM2A7BAW5JLLAUXC5KY3OIJVDUFMD2QJOMAWO2YZPDGYTV3O5UQ) context deadline exceeded ======LOGS====== Dec 16 12:43:13.728557058 (261h14m13.599640103s) [upload=229047]: Starting upload Dec 16 12:43:13.728630357 (261h14m13.599713392s) [upload=229047]: Done starting upload Dec 16 12:43:13.728720429 (261h14m13.599803464s) [upload=229047]: Uploading segments... Dec 16 12:43:13.728870865 (261h14m13.59995391s) [upload=229047]: Got next segment. Inline: false Dec 16 12:43:13.728909358 (261h14m13.599992394s) [upload=229047]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 12:43:13.728916722 (261h14m13.599999757s) [upload=229047]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 12:43:13.972957534 (261h14m13.844040579s) [seg_pos={0 0}, upload=229047]: Begin upload segment... Dec 16 12:43:13.973018189 (261h14m13.844101224s) [seg_pos={0 0}, upload=229047]: Done begin upload segment. Dec 16 12:43:13.973047195 (261h14m13.84413023s) [noise=true , address=45.91.116.10:4084 , piece_id=PIM2A7BAW5JLLAUXC5KY3OIJVDUFMD2QJOMAWO2YZPDGYTV3O5UQ, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.973077572 (261h14m13.844160607s) [noise=true , address=145.79.227.38:4979 , piece_id=ATQJMSS4OGOYOV7JHJIR5L2AXJ3JG2BDCZSH2BSMFKTCLFWBRPQQ, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.973129691 (261h14m13.844212727s) [noise=true , address=145.79.227.42:4123 , piece_id=FHBA5MWQDIO2AR2WSZKOFIOQ5CLU3KUUO6KZBI5GIYTPFSXMC6UA, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.973277132 (261h14m13.844360167s) [noise=true , address=145.79.227.15:4876 , piece_id=HTTRABRS77W63M247BPVM4355EXJ2XM5RAE4WYTU5Y57BW3KNTMA, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.974019627 (261h14m13.845102672s) [noise=true , address=145.79.227.36:4519 , piece_id=3PCQPIRPQRYNQLQ3E26DSPFEAN5OIZSPWLFNJFXG7LGSL7R5R56Q, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.974469202 (261h14m13.845552237s) [noise=true , address=145.79.227.10:4403 , piece_id=EXJQUKT3RRKVPYILF7OHM4KS65YBNJOBLZ5BDX4FQKZNPCZPLOXA, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.974574993 (261h14m13.845658038s) [noise=true , address=145.79.227.30:4762 , piece_id=43BVWF67M2D2JGE6GYLEHCNG2AWDSZAWHJD4MLUXK5SYCNY7RV2A, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.974906495 (261h14m13.84598954s) [noise=true , address=145.79.227.37:4567 , piece_id=BKXYONQBJCTHDQTOCIU4EL5DHIBFHG72ITE7DHULTGWQAOL4TGNQ, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.975608862 (261h14m13.846691907s) [noise=true , address=77.90.31.21:4877 , piece_id=HX5W43RBRV77ZVEW6PNPFJYNY42GHPP6IGM3EBYVRJOOULYTO6GQ, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.977027534 (261h14m13.848110579s) [noise=true , address=145.79.227.49:4337 , piece_id=4EBKFKOE7QQ2H3XA6ZMYVZLEIAIQI3OVIIABVNBXMNXBMJO76TVQ, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.978403523 (261h14m13.849486568s) [noise=true , address=45.91.116.8:4045 , piece_id=S5CJERIQY3GZAJXN6ZO4F7KPWG6YRIEUHZUVG32G6F2AD2IB7J6Q, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.979495323 (261h14m13.850578368s) [noise=true , address=145.79.227.23:4545 , piece_id=CMBWXQ5JQ7VMLECNCREEBSYCNEIJBLD7VURTIWE3EOAJ437255JA, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.980210935 (261h14m13.85129398s) [noise=true , address=145.79.227.45:4230 , piece_id=OXTKOHXIDLFHNWB6AGZTO46W6ISVECHVXTUMB36JC6JICYQAHPNQ, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.981410179 (261h14m13.852493214s) [noise=true , address=45.91.116.14:4190 , piece_id=CQDXY75T4WKG2LOAARKP6M4UBZ7ESH3YY2Y6TZR23OFQQBBSCK7A, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.984172609 (261h14m13.855255654s) [noise=true , address=77.90.31.10:4517 , piece_id=LCWP7QIRJEIU6ZAPN7UENDAIH2AP6PSNJBUROBVDA53TQXXUMY6A, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.984662782 (261h14m13.855745827s) [noise=true , address=145.79.227.35:4483 , piece_id=MHTKY2G6SIU2PNBESDVHCXBXHZGFR7YPDN7J3YU4CJDF5BF3VMVA, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:15.086530873 (261h14m14.957613918s) [noise=true , address=77.90.31.21:4877 , piece_id=HX5W43RBRV77ZVEW6PNPFJYNY42GHPP6IGM3EBYVRJOOULYTO6GQ, seg_pos={0 0}, upload=229047]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 16 12:43:15.669980685 (261h14m15.54106373s) [upload=229047]: Next returned nil segment Dec 16 12:43:15.669997526 (261h14m15.541080571s) [upload=229047]: Waiting for error group managing segments... Dec 16 12:43:15.903820400 (261h14m15.774903435s) [noise=true , address=77.90.31.10:4517 , piece_id=LCWP7QIRJEIU6ZAPN7UENDAIH2AP6PSNJBUROBVDA53TQXXUMY6A, seg_pos={0 0}, upload=229047]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: storage node overloaded, request limit: 2000 Dec 16 12:43:15.951902859 (261h14m15.822985904s) [noise=true , address=145.79.227.10:4403 , piece_id=EXJQUKT3RRKVPYILF7OHM4KS65YBNJOBLZ5BDX4FQKZNPCZPLOXA, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:15.973291127 (261h14m15.844374162s) [noise=true , address=145.79.227.42:4123 , piece_id=FHBA5MWQDIO2AR2WSZKOFIOQ5CLU3KUUO6KZBI5GIYTPFSXMC6UA, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:15.973360790 (261h14m15.844443835s) [noise=true , address=145.79.227.23:4545 , piece_id=CMBWXQ5JQ7VMLECNCREEBSYCNEIJBLD7VURTIWE3EOAJ437255JA, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:15.982520812 (261h14m15.853603857s) [noise=true , address=145.79.227.38:4979 , piece_id=ATQJMSS4OGOYOV7JHJIR5L2AXJ3JG2BDCZSH2BSMFKTCLFWBRPQQ, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:15.984472408 (261h14m15.855555443s) [noise=true , address=145.79.227.49:4337 , piece_id=4EBKFKOE7QQ2H3XA6ZMYVZLEIAIQI3OVIIABVNBXMNXBMJO76TVQ, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:15.986032358 (261h14m15.857115403s) [noise=true , address=145.79.227.15:4876 , piece_id=HTTRABRS77W63M247BPVM4355EXJ2XM5RAE4WYTU5Y57BW3KNTMA, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:15.986742190 (261h14m15.857825225s) [noise=true , address=145.79.227.35:4483 , piece_id=MHTKY2G6SIU2PNBESDVHCXBXHZGFR7YPDN7J3YU4CJDF5BF3VMVA, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:15.996623816 (261h14m15.867706851s) [noise=true , address=145.79.227.45:4230 , piece_id=OXTKOHXIDLFHNWB6AGZTO46W6ISVECHVXTUMB36JC6JICYQAHPNQ, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:16.051285142 (261h14m15.922368178s) [noise=true , address=145.79.227.36:4519 , piece_id=3PCQPIRPQRYNQLQ3E26DSPFEAN5OIZSPWLFNJFXG7LGSL7R5R56Q, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:16.055795661 (261h14m15.926878696s) [noise=true , address=145.79.227.37:4567 , piece_id=BKXYONQBJCTHDQTOCIU4EL5DHIBFHG72ITE7DHULTGWQAOL4TGNQ, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:16.292043703 (261h14m16.163126758s) [noise=true , address=45.91.116.14:4190 , piece_id=CQDXY75T4WKG2LOAARKP6M4UBZ7ESH3YY2Y6TZR23OFQQBBSCK7A, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:16.323293028 (261h14m16.194376073s) [noise=true , address=45.91.116.8:4045 , piece_id=S5CJERIQY3GZAJXN6ZO4F7KPWG6YRIEUHZUVG32G6F2AD2IB7J6Q, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:17.080613814 (261h14m16.951696859s) [noise=true , address=45.91.116.10:4084 , piece_id=PIM2A7BAW5JLLAUXC5KY3OIJVDUFMD2QJOMAWO2YZPDGYTV3O5UQ, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:17.080636187 (261h14m16.951719222s) [seg_pos={0 0}, upload=229047]: Segment reached optimal threshold of 13 pieces. Dec 16 12:43:17.080908635 (261h14m16.95199169s) [noise=true , address=145.79.227.30:4762 , piece_id=43BVWF67M2D2JGE6GYLEHCNG2AWDSZAWHJD4MLUXK5SYCNY7RV2A, seg_pos={0 0}, upload=229047]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Dec 16 12:43:17.080919286 (261h14m16.952002321s) [noise=true , address=145.79.227.30:4762 , piece_id=43BVWF67M2D2JGE6GYLEHCNG2AWDSZAWHJD4MLUXK5SYCNY7RV2A, seg_pos={0 0}, upload=229047]: Long tail cancellation due to optimal threshold Dec 16 12:43:17.080942920 (261h14m16.952025955s) [seg_pos={0 0}, upload=229047]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 12:43:17.081036538 (261h14m16.952119583s) [upload=229047]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 12:43:17.081043632 (261h14m16.952126677s) [upload=229047]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 12:43:17.371280229 (261h14m17.242363264s) [upload=229047]: Done uploading segments... Dec 16 12:43:17.371312330 (261h14m17.242395375s) [upload=229047]: Upload finished. err: <nil> Duration: 6.339199123s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..........................XX............................X..X] duration: 30.002452497s trace: https://jaeger.dev.storj.io/trace/6da156ed54963bae FAIL (2025-12-16 12:43:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: CQDXY75T4WKG2LOAARKP6M4UBZ7ESH3YY2Y6TZR23OFQQBBSCK7A) context deadline exceeded; piecestore: (Node ID: 1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, Piece ID: PIM2A7BAW5JLLAUXC5KY3OIJVDUFMD2QJOMAWO2YZPDGYTV3O5UQ) context deadline exceeded ======LOGS====== Dec 16 12:43:13.728557058 (261h14m13.599640103s) [upload=229047]: Starting upload Dec 16 12:43:13.728630357 (261h14m13.599713392s) [upload=229047]: Done starting upload Dec 16 12:43:13.728720429 (261h14m13.599803464s) [upload=229047]: Uploading segments... Dec 16 12:43:13.728870865 (261h14m13.59995391s) [upload=229047]: Got next segment. Inline: false Dec 16 12:43:13.728909358 (261h14m13.599992394s) [upload=229047]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 12:43:13.728916722 (261h14m13.599999757s) [upload=229047]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 12:43:13.972957534 (261h14m13.844040579s) [seg_pos={0 0}, upload=229047]: Begin upload segment... Dec 16 12:43:13.973018189 (261h14m13.844101224s) [seg_pos={0 0}, upload=229047]: Done begin upload segment. Dec 16 12:43:13.973047195 (261h14m13.84413023s) [noise=true , address=45.91.116.10:4084 , piece_id=PIM2A7BAW5JLLAUXC5KY3OIJVDUFMD2QJOMAWO2YZPDGYTV3O5UQ, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.973077572 (261h14m13.844160607s) [noise=true , address=145.79.227.38:4979 , piece_id=ATQJMSS4OGOYOV7JHJIR5L2AXJ3JG2BDCZSH2BSMFKTCLFWBRPQQ, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.973129691 (261h14m13.844212727s) [noise=true , address=145.79.227.42:4123 , piece_id=FHBA5MWQDIO2AR2WSZKOFIOQ5CLU3KUUO6KZBI5GIYTPFSXMC6UA, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.973277132 (261h14m13.844360167s) [noise=true , address=145.79.227.15:4876 , piece_id=HTTRABRS77W63M247BPVM4355EXJ2XM5RAE4WYTU5Y57BW3KNTMA, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.974019627 (261h14m13.845102672s) [noise=true , address=145.79.227.36:4519 , piece_id=3PCQPIRPQRYNQLQ3E26DSPFEAN5OIZSPWLFNJFXG7LGSL7R5R56Q, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.974469202 (261h14m13.845552237s) [noise=true , address=145.79.227.10:4403 , piece_id=EXJQUKT3RRKVPYILF7OHM4KS65YBNJOBLZ5BDX4FQKZNPCZPLOXA, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.974574993 (261h14m13.845658038s) [noise=true , address=145.79.227.30:4762 , piece_id=43BVWF67M2D2JGE6GYLEHCNG2AWDSZAWHJD4MLUXK5SYCNY7RV2A, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.974906495 (261h14m13.84598954s) [noise=true , address=145.79.227.37:4567 , piece_id=BKXYONQBJCTHDQTOCIU4EL5DHIBFHG72ITE7DHULTGWQAOL4TGNQ, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.975608862 (261h14m13.846691907s) [noise=true , address=77.90.31.21:4877 , piece_id=HX5W43RBRV77ZVEW6PNPFJYNY42GHPP6IGM3EBYVRJOOULYTO6GQ, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.977027534 (261h14m13.848110579s) [noise=true , address=145.79.227.49:4337 , piece_id=4EBKFKOE7QQ2H3XA6ZMYVZLEIAIQI3OVIIABVNBXMNXBMJO76TVQ, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.978403523 (261h14m13.849486568s) [noise=true , address=45.91.116.8:4045 , piece_id=S5CJERIQY3GZAJXN6ZO4F7KPWG6YRIEUHZUVG32G6F2AD2IB7J6Q, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.979495323 (261h14m13.850578368s) [noise=true , address=145.79.227.23:4545 , piece_id=CMBWXQ5JQ7VMLECNCREEBSYCNEIJBLD7VURTIWE3EOAJ437255JA, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.980210935 (261h14m13.85129398s) [noise=true , address=145.79.227.45:4230 , piece_id=OXTKOHXIDLFHNWB6AGZTO46W6ISVECHVXTUMB36JC6JICYQAHPNQ, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.981410179 (261h14m13.852493214s) [noise=true , address=45.91.116.14:4190 , piece_id=CQDXY75T4WKG2LOAARKP6M4UBZ7ESH3YY2Y6TZR23OFQQBBSCK7A, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.984172609 (261h14m13.855255654s) [noise=true , address=77.90.31.10:4517 , piece_id=LCWP7QIRJEIU6ZAPN7UENDAIH2AP6PSNJBUROBVDA53TQXXUMY6A, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.984662782 (261h14m13.855745827s) [noise=true , address=145.79.227.35:4483 , piece_id=MHTKY2G6SIU2PNBESDVHCXBXHZGFR7YPDN7J3YU4CJDF5BF3VMVA, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:15.086530873 (261h14m14.957613918s) [noise=true , address=77.90.31.21:4877 , piece_id=HX5W43RBRV77ZVEW6PNPFJYNY42GHPP6IGM3EBYVRJOOULYTO6GQ, seg_pos={0 0}, upload=229047]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 16 12:43:15.669980685 (261h14m15.54106373s) [upload=229047]: Next returned nil segment Dec 16 12:43:15.669997526 (261h14m15.541080571s) [upload=229047]: Waiting for error group managing segments... Dec 16 12:43:15.903820400 (261h14m15.774903435s) [noise=true , address=77.90.31.10:4517 , piece_id=LCWP7QIRJEIU6ZAPN7UENDAIH2AP6PSNJBUROBVDA53TQXXUMY6A, seg_pos={0 0}, upload=229047]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: storage node overloaded, request limit: 2000 Dec 16 12:43:15.951902859 (261h14m15.822985904s) [noise=true , address=145.79.227.10:4403 , piece_id=EXJQUKT3RRKVPYILF7OHM4KS65YBNJOBLZ5BDX4FQKZNPCZPLOXA, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:15.973291127 (261h14m15.844374162s) [noise=true , address=145.79.227.42:4123 , piece_id=FHBA5MWQDIO2AR2WSZKOFIOQ5CLU3KUUO6KZBI5GIYTPFSXMC6UA, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:15.973360790 (261h14m15.844443835s) [noise=true , address=145.79.227.23:4545 , piece_id=CMBWXQ5JQ7VMLECNCREEBSYCNEIJBLD7VURTIWE3EOAJ437255JA, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:15.982520812 (261h14m15.853603857s) [noise=true , address=145.79.227.38:4979 , piece_id=ATQJMSS4OGOYOV7JHJIR5L2AXJ3JG2BDCZSH2BSMFKTCLFWBRPQQ, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:15.984472408 (261h14m15.855555443s) [noise=true , address=145.79.227.49:4337 , piece_id=4EBKFKOE7QQ2H3XA6ZMYVZLEIAIQI3OVIIABVNBXMNXBMJO76TVQ, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:15.986032358 (261h14m15.857115403s) [noise=true , address=145.79.227.15:4876 , piece_id=HTTRABRS77W63M247BPVM4355EXJ2XM5RAE4WYTU5Y57BW3KNTMA, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:15.986742190 (261h14m15.857825225s) [noise=true , address=145.79.227.35:4483 , piece_id=MHTKY2G6SIU2PNBESDVHCXBXHZGFR7YPDN7J3YU4CJDF5BF3VMVA, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:15.996623816 (261h14m15.867706851s) [noise=true , address=145.79.227.45:4230 , piece_id=OXTKOHXIDLFHNWB6AGZTO46W6ISVECHVXTUMB36JC6JICYQAHPNQ, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:16.051285142 (261h14m15.922368178s) [noise=true , address=145.79.227.36:4519 , piece_id=3PCQPIRPQRYNQLQ3E26DSPFEAN5OIZSPWLFNJFXG7LGSL7R5R56Q, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:16.055795661 (261h14m15.926878696s) [noise=true , address=145.79.227.37:4567 , piece_id=BKXYONQBJCTHDQTOCIU4EL5DHIBFHG72ITE7DHULTGWQAOL4TGNQ, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:16.292043703 (261h14m16.163126758s) [noise=true , address=45.91.116.14:4190 , piece_id=CQDXY75T4WKG2LOAARKP6M4UBZ7ESH3YY2Y6TZR23OFQQBBSCK7A, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:16.323293028 (261h14m16.194376073s) [noise=true , address=45.91.116.8:4045 , piece_id=S5CJERIQY3GZAJXN6ZO4F7KPWG6YRIEUHZUVG32G6F2AD2IB7J6Q, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:17.080613814 (261h14m16.951696859s) [noise=true , address=45.91.116.10:4084 , piece_id=PIM2A7BAW5JLLAUXC5KY3OIJVDUFMD2QJOMAWO2YZPDGYTV3O5UQ, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:17.080636187 (261h14m16.951719222s) [seg_pos={0 0}, upload=229047]: Segment reached optimal threshold of 13 pieces. Dec 16 12:43:17.080908635 (261h14m16.95199169s) [noise=true , address=145.79.227.30:4762 , piece_id=43BVWF67M2D2JGE6GYLEHCNG2AWDSZAWHJD4MLUXK5SYCNY7RV2A, seg_pos={0 0}, upload=229047]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Dec 16 12:43:17.080919286 (261h14m16.952002321s) [noise=true , address=145.79.227.30:4762 , piece_id=43BVWF67M2D2JGE6GYLEHCNG2AWDSZAWHJD4MLUXK5SYCNY7RV2A, seg_pos={0 0}, upload=229047]: Long tail cancellation due to optimal threshold Dec 16 12:43:17.080942920 (261h14m16.952025955s) [seg_pos={0 0}, upload=229047]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 12:43:17.081036538 (261h14m16.952119583s) [upload=229047]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 12:43:17.081043632 (261h14m16.952126677s) [upload=229047]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 12:43:17.371280229 (261h14m17.242363264s) [upload=229047]: Done uploading segments... Dec 16 12:43:17.371312330 (261h14m17.242395375s) [upload=229047]: Upload finished. err: <nil> Duration: 30.002476523s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 22 out of 60 is failed (threshold=6) [...X.XX.......X....X...XXX......X.X...X........X.XXXXXXXXXX.] duration: 3.686608087s trace: https://jaeger.dev.storj.io/trace/157fe60608367bfb FAIL (2025-12-16 12:43:30 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.686640678s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 22 out of 60 is failed (threshold=6) [....X.XX.......X....X...XXX......X.X...X........X.XXXXXXXXXX] duration: 5.001444769s trace: https://jaeger.dev.storj.io/trace/157fe60608367bfb FAIL (2025-12-16 12:43:16 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001470849s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...........................XX............................X..] duration: 5.756307266s trace: https://jaeger.dev.storj.io/trace/438d408263f71c1c FAIL (2025-12-16 12:42:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 16 12:41:02.374128620 (261h12m2.245211665s) [upload=229019]: Starting upload Dec 16 12:41:02.374510256 (261h12m2.245593301s) [upload=229019]: Done starting upload Dec 16 12:41:02.374735605 (261h12m2.24581866s) [upload=229019]: Uploading segments... Dec 16 12:41:02.374845614 (261h12m2.245928659s) [upload=229019]: Got next segment. Inline: false Dec 16 12:41:02.374884409 (261h12m2.245967444s) [upload=229019]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 12:41:02.374891242 (261h12m2.245974287s) [upload=229019]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 12:41:02.618261323 (261h12m2.489344368s) [seg_pos={0 0}, upload=229019]: Begin upload segment... Dec 16 12:41:02.618345263 (261h12m2.489428308s) [seg_pos={0 0}, upload=229019]: Done begin upload segment. Dec 16 12:41:02.618384808 (261h12m2.489467853s) [noise=true , address=77.90.31.17:4773 , piece_id=ENBWT2AN2UT67GOXDGFOCYSB6MUVWFCED6HH5DG7RDJA6BOA2ZNA, seg_pos={0 0}, upload=229019]: Uploading piece... Dec 16 12:41:02.618465531 (261h12m2.489548576s) [noise=true , address=145.79.227.50:4374 , piece_id=NDRBX4YXQ5ZGVX2EFZNTZODGZD3F6OVBOKTIKEQFU3IMGSW6XHRQ, seg_pos={0 0}, upload=229019]: Uploading piece... Dec 16 12:41:02.619174953 (261h12m2.490257998s) [noise=true , address=145.79.227.52:4446 , piece_id=D6C54FW7JTL54Z7K7H3EVMS7VYDZ76RNQJD3VRJXWJYVTC6CUS2A, seg_pos={0 0}, upload=229019]: Uploading piece... Dec 16 12:41:02.619590514 (261h12m2.490673559s) [noise=true , address=145.79.227.40:4051 , piece_id=4LLC5QT5VD3Q7KTUZUNPRN7H2HS7KPAI53VKLCH2L3FSLFQJD7SA, seg_pos={0 0}, upload=229019]: Uploading piece... Dec 16 12:41:02.619700574 (261h12m2.490783739s) [noise=true , address=145.79.227.42:4121 , piece_id=AMSPQNXERGDDA4FX6U3V3TQ5KGQ5P2CHFJUBT3TJOIGYXC4H7BTQ, seg_pos={0 0}, upload=229019]: Uploading piece... Dec 16 12:41:02.619941132 (261h12m2.491024177s) [noise=true , address=145.79.227.23:4545 , piece_id=HSM2EF4XUIRY5S66UYUNYCGDRYZLKXP5KNREZ637K3267KEQ6AKA, seg_pos={0 0}, upload=229019]: Uploading piece... Dec 16 12:41:02.620353237 (261h12m2.491436292s) [noise=true , address=145.79.227.2:4114 , piece_id=OOHN3NOCIWV4PJQFP6PP5HNSE7AMQNIYUTNE3GUZWGZNNBIDJLVA, seg_pos={0 0}, upload=229019]: Uploading piece... Dec 16 12:41:02.620873718 (261h12m2.491956763s) [noise=true , address=77.90.31.13:4627 , piece_id=UVT3WIYHWEAODXXQEOCL44ITURWNPKQCOBJGSAOGE22FZFGW7KIA, seg_pos={0 0}, upload=229019]: Uploading piece... Dec 16 12:41:02.621508055 (261h12m2.4925911s) [noise=true , address=145.79.227.37:4565 , piece_id=LYV7QPUIRN6GBHOA3AZQZEGD3OBLV7FHJ5VQNCQNOCRR7XYJGJKA, seg_pos={0 0}, upload=229019]: Uploading piece... Dec 16 12:41:02.622063874 (261h12m2.493146929s) [noise=true , address=45.91.116.4:4939 , piece_id=33J2XYXK3HJCKKEKNYF2ZFFXVLN7SDKKPDODEKJBCB4H2SPFEI3Q, seg_pos={0 0}, upload=229019]: Uploading piece... Dec 16 12:41:02.622713400 (261h12m2.493796445s) [noise=true , address=45.91.116.8:4046 , piece_id=H2EMKIPYJ5LX4YAJYL5LL3TNANU7XC3QMZZDTSAS4VF4RKXBR45A, seg_pos={0 0}, upload=229019]: Uploading piece... Dec 16 12:41:02.623064349 (261h12m2.494147404s) [noise=true , address=145.79.227.31:4797 , piece_id=HEFSDSKFEPWBXNX6UUODOLJY6ELLCCZPWEMAEDY2YJPZHC76QU6Q, seg_pos={0 0}, upload=229019]: Uploading piece... Dec 16 12:41:02.623515127 (261h12m2.494598202s) [noise=true , address=45.91.116.5:4300 , piece_id=KR7FEGNXVBVVQNPERSU3IJBAJCKSMFSZF3K4RSQF34EFBVB2Y6TQ, seg_pos={0 0}, upload=229019]: Uploading piece... Dec 16 12:41:02.623962238 (261h12m2.495045273s) [noise=true , address=145.79.227.46:4603 , piece_id=33ZVWVMTLXVUVXONZ5XF25LBPCNRJSLW2VXNYZIAVFRT3AIX24DQ, seg_pos={0 0}, upload=229019]: Uploading piece... Dec 16 12:41:02.627737227 (261h12m2.498820272s) [noise=true , address=77.90.31.19:4844 , piece_id=PL77RE6OFQMUBS2QE2CVANQF2B4YXX2ZURW4VTOW56ONTXYVNHPA, seg_pos={0 0}, upload=229019]: Uploading piece... Dec 16 12:41:02.628421890 (261h12m2.499504925s) [noise=true , address=145.79.227.51:4411 , piece_id=2XSX4FFN7QDB42AMYAWK7RRW2ZIVUJOIO4HXRIRH5PJA4E5PIQ6A, seg_pos={0 0}, upload=229019]: Uploading piece... Dec 16 12:41:04.307455098 (261h12m4.178538143s) [upload=229019]: Next returned nil segment Dec 16 12:41:04.307483461 (261h12m4.178566506s) [upload=229019]: Waiting for error group managing segments... Dec 16 12:41:04.616131484 (261h12m4.487214529s) [noise=true , address=145.79.227.23:4545 , piece_id=HSM2EF4XUIRY5S66UYUNYCGDRYZLKXP5KNREZ637K3267KEQ6AKA, seg_pos={0 0}, upload=229019]: Done uploading piece. err: <nil> Dec 16 12:41:04.618006805 (261h12m4.48908985s) [noise=true , address=145.79.227.52:4446 , piece_id=D6C54FW7JTL54Z7K7H3EVMS7VYDZ76RNQJD3VRJXWJYVTC6CUS2A, seg_pos={0 0}, upload=229019]: Done uploading piece. err: <nil> Dec 16 12:41:04.618126533 (261h12m4.489209578s) [noise=true , address=145.79.227.42:4121 , piece_id=AMSPQNXERGDDA4FX6U3V3TQ5KGQ5P2CHFJUBT3TJOIGYXC4H7BTQ, seg_pos={0 0}, upload=229019]: Done uploading piece. err: <nil> Dec 16 12:41:04.620233455 (261h12m4.49131649s) [noise=true , address=145.79.227.37:4565 , piece_id=LYV7QPUIRN6GBHOA3AZQZEGD3OBLV7FHJ5VQNCQNOCRR7XYJGJKA, seg_pos={0 0}, upload=229019]: Done uploading piece. err: <nil> Dec 16 12:41:04.633476863 (261h12m4.504559898s) [noise=true , address=145.79.227.2:4114 , piece_id=OOHN3NOCIWV4PJQFP6PP5HNSE7AMQNIYUTNE3GUZWGZNNBIDJLVA, seg_pos={0 0}, upload=229019]: Done uploading piece. err: <nil> Dec 16 12:41:04.640074275 (261h12m4.51115732s) [noise=true , address=145.79.227.50:4374 , piece_id=NDRBX4YXQ5ZGVX2EFZNTZODGZD3F6OVBOKTIKEQFU3IMGSW6XHRQ, seg_pos={0 0}, upload=229019]: Done uploading piece. err: <nil> Dec 16 12:41:04.640631055 (261h12m4.5117141s) [noise=true , address=145.79.227.46:4603 , piece_id=33ZVWVMTLXVUVXONZ5XF25LBPCNRJSLW2VXNYZIAVFRT3AIX24DQ, seg_pos={0 0}, upload=229019]: Done uploading piece. err: <nil> Dec 16 12:41:04.654684656 (261h12m4.525767691s) [noise=true , address=145.79.227.31:4797 , piece_id=HEFSDSKFEPWBXNX6UUODOLJY6ELLCCZPWEMAEDY2YJPZHC76QU6Q, seg_pos={0 0}, upload=229019]: Done uploading piece. err: <nil> Dec 16 12:41:04.687122594 (261h12m4.558205639s) [noise=true , address=145.79.227.51:4411 , piece_id=2XSX4FFN7QDB42AMYAWK7RRW2ZIVUJOIO4HXRIRH5PJA4E5PIQ6A, seg_pos={0 0}, upload=229019]: Done uploading piece. err: <nil> Dec 16 12:41:04.699192387 (261h12m4.570275422s) [noise=true , address=145.79.227.40:4051 , piece_id=4LLC5QT5VD3Q7KTUZUNPRN7H2HS7KPAI53VKLCH2L3FSLFQJD7SA, seg_pos={0 0}, upload=229019]: Done uploading piece. err: <nil> Dec 16 12:41:04.827971536 (261h12m4.699054571s) [noise=true , address=45.91.116.8:4046 , piece_id=H2EMKIPYJ5LX4YAJYL5LL3TNANU7XC3QMZZDTSAS4VF4RKXBR45A, seg_pos={0 0}, upload=229019]: Done uploading piece. err: <nil> Dec 16 12:41:04.891714144 (261h12m4.762797189s) [noise=true , address=77.90.31.17:4773 , piece_id=ENBWT2AN2UT67GOXDGFOCYSB6MUVWFCED6HH5DG7RDJA6BOA2ZNA, seg_pos={0 0}, upload=229019]: Done uploading piece. err: <nil> Dec 16 12:41:04.897785625 (261h12m4.76886867s) [noise=true , address=77.90.31.19:4844 , piece_id=PL77RE6OFQMUBS2QE2CVANQF2B4YXX2ZURW4VTOW56ONTXYVNHPA, seg_pos={0 0}, upload=229019]: Done uploading piece. err: <nil> Dec 16 12:41:04.897858313 (261h12m4.768941348s) [seg_pos={0 0}, upload=229019]: Segment reached optimal threshold of 13 pieces. Dec 16 12:41:04.898181489 (261h12m4.769264534s) [noise=true , address=45.91.116.5:4300 , piece_id=KR7FEGNXVBVVQNPERSU3IJBAJCKSMFSZF3K4RSQF34EFBVB2Y6TQ, seg_pos={0 0}, upload=229019]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF Dec 16 12:41:04.898192810 (261h12m4.769275855s) [noise=true , address=45.91.116.5:4300 , piece_id=KR7FEGNXVBVVQNPERSU3IJBAJCKSMFSZF3K4RSQF34EFBVB2Y6TQ, seg_pos={0 0}, upload=229019]: Long tail cancellation due to optimal threshold Dec 16 12:41:04.898618681 (261h12m4.769701726s) [noise=true , address=45.91.116.4:4939 , piece_id=33J2XYXK3HJCKKEKNYF2ZFFXVLN7SDKKPDODEKJBCB4H2SPFEI3Q, seg_pos={0 0}, upload=229019]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Dec 16 12:41:04.898657345 (261h12m4.76974039s) [noise=true , address=45.91.116.4:4939 , piece_id=33J2XYXK3HJCKKEKNYF2ZFFXVLN7SDKKPDODEKJBCB4H2SPFEI3Q, seg_pos={0 0}, upload=229019]: Long tail cancellation due to optimal threshold Dec 16 12:41:04.898778977 (261h12m4.769862022s) [noise=true , address=77.90.31.13:4627 , piece_id=UVT3WIYHWEAODXXQEOCL44ITURWNPKQCOBJGSAOGE22FZFGW7KIA, seg_pos={0 0}, upload=229019]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: read tcp 188.166.225.170:41726->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 16 12:41:04.898789476 (261h12m4.769872511s) [noise=true , address=77.90.31.13:4627 , piece_id=UVT3WIYHWEAODXXQEOCL44ITURWNPKQCOBJGSAOGE22FZFGW7KIA, seg_pos={0 0}, upload=229019]: Long tail cancellation due to optimal threshold Dec 16 12:41:04.898815366 (261h12m4.769898411s) [seg_pos={0 0}, upload=229019]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 12:41:04.898903884 (261h12m4.769986939s) [upload=229019]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 12:41:04.898911098 (261h12m4.769994144s) [upload=229019]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 12:41:05.201351762 (261h12m5.072434807s) [upload=229019]: Done uploading segments... Dec 16 12:41:05.201374225 (261h12m5.07245726s) [upload=229019]: Upload finished. err: <nil> Duration: 5.756337234s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 22 out of 60 is failed (threshold=6) [X....X.XX.......X....X...XXX......X.X...X........X.XXXXXXXXX] duration: 5.001351169s trace: https://jaeger.dev.storj.io/trace/263f61ad30d705e5 FAIL (2025-12-16 12:42:03 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001378411s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [............................XX............................X.] duration: 5.822078486s trace: https://jaeger.dev.storj.io/trace/438d408263f71c1c FAIL (2025-12-16 12:41:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 16 12:41:02.374128620 (261h12m2.245211665s) [upload=229019]: Starting upload Dec 16 12:41:02.374510256 (261h12m2.245593301s) [upload=229019]: Done starting upload Dec 16 12:41:02.374735605 (261h12m2.24581866s) [upload=229019]: Uploading segments... Dec 16 12:41:02.374845614 (261h12m2.245928659s) [upload=229019]: Got next segment. Inline: false Dec 16 12:41:02.374884409 (261h12m2.245967444s) [upload=229019]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 12:41:02.374891242 (261h12m2.245974287s) [upload=229019]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 12:41:02.618261323 (261h12m2.489344368s) [seg_pos={0 0}, upload=229019]: Begin upload segment... Dec 16 12:41:02.618345263 (261h12m2.489428308s) [seg_pos={0 0}, upload=229019]: Done begin upload segment. Dec 16 12:41:02.618384808 (261h12m2.489467853s) [noise=true , address=77.90.31.17:4773 , piece_id=ENBWT2AN2UT67GOXDGFOCYSB6MUVWFCED6HH5DG7RDJA6BOA2ZNA, seg_pos={0 0}, upload=229019]: Uploading piece... Dec 16 12:41:02.618465531 (261h12m2.489548576s) [noise=true , address=145.79.227.50:4374 , piece_id=NDRBX4YXQ5ZGVX2EFZNTZODGZD3F6OVBOKTIKEQFU3IMGSW6XHRQ, seg_pos={0 0}, upload=229019]: Uploading piece... Dec 16 12:41:02.619174953 (261h12m2.490257998s) [noise=true , address=145.79.227.52:4446 , piece_id=D6C54FW7JTL54Z7K7H3EVMS7VYDZ76RNQJD3VRJXWJYVTC6CUS2A, seg_pos={0 0}, upload=229019]: Uploading piece... Dec 16 12:41:02.619590514 (261h12m2.490673559s) [noise=true , address=145.79.227.40:4051 , piece_id=4LLC5QT5VD3Q7KTUZUNPRN7H2HS7KPAI53VKLCH2L3FSLFQJD7SA, seg_pos={0 0}, upload=229019]: Uploading piece... Dec 16 12:41:02.619700574 (261h12m2.490783739s) [noise=true , address=145.79.227.42:4121 , piece_id=AMSPQNXERGDDA4FX6U3V3TQ5KGQ5P2CHFJUBT3TJOIGYXC4H7BTQ, seg_pos={0 0}, upload=229019]: Uploading piece... Dec 16 12:41:02.619941132 (261h12m2.491024177s) [noise=true , address=145.79.227.23:4545 , piece_id=HSM2EF4XUIRY5S66UYUNYCGDRYZLKXP5KNREZ637K3267KEQ6AKA, seg_pos={0 0}, upload=229019]: Uploading piece... Dec 16 12:41:02.620353237 (261h12m2.491436292s) [noise=true , address=145.79.227.2:4114 , piece_id=OOHN3NOCIWV4PJQFP6PP5HNSE7AMQNIYUTNE3GUZWGZNNBIDJLVA, seg_pos={0 0}, upload=229019]: Uploading piece... Dec 16 12:41:02.620873718 (261h12m2.491956763s) [noise=true , address=77.90.31.13:4627 , piece_id=UVT3WIYHWEAODXXQEOCL44ITURWNPKQCOBJGSAOGE22FZFGW7KIA, seg_pos={0 0}, upload=229019]: Uploading piece... Dec 16 12:41:02.621508055 (261h12m2.4925911s) [noise=true , address=145.79.227.37:4565 , piece_id=LYV7QPUIRN6GBHOA3AZQZEGD3OBLV7FHJ5VQNCQNOCRR7XYJGJKA, seg_pos={0 0}, upload=229019]: Uploading piece... Dec 16 12:41:02.622063874 (261h12m2.493146929s) [noise=true , address=45.91.116.4:4939 , piece_id=33J2XYXK3HJCKKEKNYF2ZFFXVLN7SDKKPDODEKJBCB4H2SPFEI3Q, seg_pos={0 0}, upload=229019]: Uploading piece... Dec 16 12:41:02.622713400 (261h12m2.493796445s) [noise=true , address=45.91.116.8:4046 , piece_id=H2EMKIPYJ5LX4YAJYL5LL3TNANU7XC3QMZZDTSAS4VF4RKXBR45A, seg_pos={0 0}, upload=229019]: Uploading piece... Dec 16 12:41:02.623064349 (261h12m2.494147404s) [noise=true , address=145.79.227.31:4797 , piece_id=HEFSDSKFEPWBXNX6UUODOLJY6ELLCCZPWEMAEDY2YJPZHC76QU6Q, seg_pos={0 0}, upload=229019]: Uploading piece... Dec 16 12:41:02.623515127 (261h12m2.494598202s) [noise=true , address=45.91.116.5:4300 , piece_id=KR7FEGNXVBVVQNPERSU3IJBAJCKSMFSZF3K4RSQF34EFBVB2Y6TQ, seg_pos={0 0}, upload=229019]: Uploading piece... Dec 16 12:41:02.623962238 (261h12m2.495045273s) [noise=true , address=145.79.227.46:4603 , piece_id=33ZVWVMTLXVUVXONZ5XF25LBPCNRJSLW2VXNYZIAVFRT3AIX24DQ, seg_pos={0 0}, upload=229019]: Uploading piece... Dec 16 12:41:02.627737227 (261h12m2.498820272s) [noise=true , address=77.90.31.19:4844 , piece_id=PL77RE6OFQMUBS2QE2CVANQF2B4YXX2ZURW4VTOW56ONTXYVNHPA, seg_pos={0 0}, upload=229019]: Uploading piece... Dec 16 12:41:02.628421890 (261h12m2.499504925s) [noise=true , address=145.79.227.51:4411 , piece_id=2XSX4FFN7QDB42AMYAWK7RRW2ZIVUJOIO4HXRIRH5PJA4E5PIQ6A, seg_pos={0 0}, upload=229019]: Uploading piece... Dec 16 12:41:04.307455098 (261h12m4.178538143s) [upload=229019]: Next returned nil segment Dec 16 12:41:04.307483461 (261h12m4.178566506s) [upload=229019]: Waiting for error group managing segments... Dec 16 12:41:04.616131484 (261h12m4.487214529s) [noise=true , address=145.79.227.23:4545 , piece_id=HSM2EF4XUIRY5S66UYUNYCGDRYZLKXP5KNREZ637K3267KEQ6AKA, seg_pos={0 0}, upload=229019]: Done uploading piece. err: <nil> Dec 16 12:41:04.618006805 (261h12m4.48908985s) [noise=true , address=145.79.227.52:4446 , piece_id=D6C54FW7JTL54Z7K7H3EVMS7VYDZ76RNQJD3VRJXWJYVTC6CUS2A, seg_pos={0 0}, upload=229019]: Done uploading piece. err: <nil> Dec 16 12:41:04.618126533 (261h12m4.489209578s) [noise=true , address=145.79.227.42:4121 , piece_id=AMSPQNXERGDDA4FX6U3V3TQ5KGQ5P2CHFJUBT3TJOIGYXC4H7BTQ, seg_pos={0 0}, upload=229019]: Done uploading piece. err: <nil> Dec 16 12:41:04.620233455 (261h12m4.49131649s) [noise=true , address=145.79.227.37:4565 , piece_id=LYV7QPUIRN6GBHOA3AZQZEGD3OBLV7FHJ5VQNCQNOCRR7XYJGJKA, seg_pos={0 0}, upload=229019]: Done uploading piece. err: <nil> Dec 16 12:41:04.633476863 (261h12m4.504559898s) [noise=true , address=145.79.227.2:4114 , piece_id=OOHN3NOCIWV4PJQFP6PP5HNSE7AMQNIYUTNE3GUZWGZNNBIDJLVA, seg_pos={0 0}, upload=229019]: Done uploading piece. err: <nil> Dec 16 12:41:04.640074275 (261h12m4.51115732s) [noise=true , address=145.79.227.50:4374 , piece_id=NDRBX4YXQ5ZGVX2EFZNTZODGZD3F6OVBOKTIKEQFU3IMGSW6XHRQ, seg_pos={0 0}, upload=229019]: Done uploading piece. err: <nil> Dec 16 12:41:04.640631055 (261h12m4.5117141s) [noise=true , address=145.79.227.46:4603 , piece_id=33ZVWVMTLXVUVXONZ5XF25LBPCNRJSLW2VXNYZIAVFRT3AIX24DQ, seg_pos={0 0}, upload=229019]: Done uploading piece. err: <nil> Dec 16 12:41:04.654684656 (261h12m4.525767691s) [noise=true , address=145.79.227.31:4797 , piece_id=HEFSDSKFEPWBXNX6UUODOLJY6ELLCCZPWEMAEDY2YJPZHC76QU6Q, seg_pos={0 0}, upload=229019]: Done uploading piece. err: <nil> Dec 16 12:41:04.687122594 (261h12m4.558205639s) [noise=true , address=145.79.227.51:4411 , piece_id=2XSX4FFN7QDB42AMYAWK7RRW2ZIVUJOIO4HXRIRH5PJA4E5PIQ6A, seg_pos={0 0}, upload=229019]: Done uploading piece. err: <nil> Dec 16 12:41:04.699192387 (261h12m4.570275422s) [noise=true , address=145.79.227.40:4051 , piece_id=4LLC5QT5VD3Q7KTUZUNPRN7H2HS7KPAI53VKLCH2L3FSLFQJD7SA, seg_pos={0 0}, upload=229019]: Done uploading piece. err: <nil> Dec 16 12:41:04.827971536 (261h12m4.699054571s) [noise=true , address=45.91.116.8:4046 , piece_id=H2EMKIPYJ5LX4YAJYL5LL3TNANU7XC3QMZZDTSAS4VF4RKXBR45A, seg_pos={0 0}, upload=229019]: Done uploading piece. err: <nil> Dec 16 12:41:04.891714144 (261h12m4.762797189s) [noise=true , address=77.90.31.17:4773 , piece_id=ENBWT2AN2UT67GOXDGFOCYSB6MUVWFCED6HH5DG7RDJA6BOA2ZNA, seg_pos={0 0}, upload=229019]: Done uploading piece. err: <nil> Dec 16 12:41:04.897785625 (261h12m4.76886867s) [noise=true , address=77.90.31.19:4844 , piece_id=PL77RE6OFQMUBS2QE2CVANQF2B4YXX2ZURW4VTOW56ONTXYVNHPA, seg_pos={0 0}, upload=229019]: Done uploading piece. err: <nil> Dec 16 12:41:04.897858313 (261h12m4.768941348s) [seg_pos={0 0}, upload=229019]: Segment reached optimal threshold of 13 pieces. Dec 16 12:41:04.898181489 (261h12m4.769264534s) [noise=true , address=45.91.116.5:4300 , piece_id=KR7FEGNXVBVVQNPERSU3IJBAJCKSMFSZF3K4RSQF34EFBVB2Y6TQ, seg_pos={0 0}, upload=229019]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF Dec 16 12:41:04.898192810 (261h12m4.769275855s) [noise=true , address=45.91.116.5:4300 , piece_id=KR7FEGNXVBVVQNPERSU3IJBAJCKSMFSZF3K4RSQF34EFBVB2Y6TQ, seg_pos={0 0}, upload=229019]: Long tail cancellation due to optimal threshold Dec 16 12:41:04.898618681 (261h12m4.769701726s) [noise=true , address=45.91.116.4:4939 , piece_id=33J2XYXK3HJCKKEKNYF2ZFFXVLN7SDKKPDODEKJBCB4H2SPFEI3Q, seg_pos={0 0}, upload=229019]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Dec 16 12:41:04.898657345 (261h12m4.76974039s) [noise=true , address=45.91.116.4:4939 , piece_id=33J2XYXK3HJCKKEKNYF2ZFFXVLN7SDKKPDODEKJBCB4H2SPFEI3Q, seg_pos={0 0}, upload=229019]: Long tail cancellation due to optimal threshold Dec 16 12:41:04.898778977 (261h12m4.769862022s) [noise=true , address=77.90.31.13:4627 , piece_id=UVT3WIYHWEAODXXQEOCL44ITURWNPKQCOBJGSAOGE22FZFGW7KIA, seg_pos={0 0}, upload=229019]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: read tcp 188.166.225.170:41726->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 16 12:41:04.898789476 (261h12m4.769872511s) [noise=true , address=77.90.31.13:4627 , piece_id=UVT3WIYHWEAODXXQEOCL44ITURWNPKQCOBJGSAOGE22FZFGW7KIA, seg_pos={0 0}, upload=229019]: Long tail cancellation due to optimal threshold Dec 16 12:41:04.898815366 (261h12m4.769898411s) [seg_pos={0 0}, upload=229019]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 12:41:04.898903884 (261h12m4.769986939s) [upload=229019]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 12:41:04.898911098 (261h12m4.769994144s) [upload=229019]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 12:41:05.201351762 (261h12m5.072434807s) [upload=229019]: Done uploading segments... Dec 16 12:41:05.201374225 (261h12m5.07245726s) [upload=229019]: Upload finished. err: <nil> Duration: 5.822104075s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.............................XX............................X] duration: 30.003171464s trace: https://jaeger.dev.storj.io/trace/438d408263f71c1c FAIL (2025-12-16 12:41:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 16 12:41:02.374128620 (261h12m2.245211665s) [upload=229019]: Starting upload Dec 16 12:41:02.374510256 (261h12m2.245593301s) [upload=229019]: Done starting upload Dec 16 12:41:02.374735605 (261h12m2.24581866s) [upload=229019]: Uploading segments... Dec 16 12:41:02.374845614 (261h12m2.245928659s) [upload=229019]: Got next segment. Inline: false Dec 16 12:41:02.374884409 (261h12m2.245967444s) [upload=229019]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 12:41:02.374891242 (261h12m2.245974287s) [upload=229019]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 12:41:02.618261323 (261h12m2.489344368s) [seg_pos={0 0}, upload=229019]: Begin upload segment... Dec 16 12:41:02.618345263 (261h12m2.489428308s) [seg_pos={0 0}, upload=229019]: Done begin upload segment. Dec 16 12:41:02.618384808 (261h12m2.489467853s) [noise=true , address=77.90.31.17:4773 , piece_id=ENBWT2AN2UT67GOXDGFOCYSB6MUVWFCED6HH5DG7RDJA6BOA2ZNA, seg_pos={0 0}, upload=229019]: Uploading piece... Dec 16 12:41:02.618465531 (261h12m2.489548576s) [noise=true , address=145.79.227.50:4374 , piece_id=NDRBX4YXQ5ZGVX2EFZNTZODGZD3F6OVBOKTIKEQFU3IMGSW6XHRQ, seg_pos={0 0}, upload=229019]: Uploading piece... Dec 16 12:41:02.619174953 (261h12m2.490257998s) [noise=true , address=145.79.227.52:4446 , piece_id=D6C54FW7JTL54Z7K7H3EVMS7VYDZ76RNQJD3VRJXWJYVTC6CUS2A, seg_pos={0 0}, upload=229019]: Uploading piece... Dec 16 12:41:02.619590514 (261h12m2.490673559s) [noise=true , address=145.79.227.40:4051 , piece_id=4LLC5QT5VD3Q7KTUZUNPRN7H2HS7KPAI53VKLCH2L3FSLFQJD7SA, seg_pos={0 0}, upload=229019]: Uploading piece... Dec 16 12:41:02.619700574 (261h12m2.490783739s) [noise=true , address=145.79.227.42:4121 , piece_id=AMSPQNXERGDDA4FX6U3V3TQ5KGQ5P2CHFJUBT3TJOIGYXC4H7BTQ, seg_pos={0 0}, upload=229019]: Uploading piece... Dec 16 12:41:02.619941132 (261h12m2.491024177s) [noise=true , address=145.79.227.23:4545 , piece_id=HSM2EF4XUIRY5S66UYUNYCGDRYZLKXP5KNREZ637K3267KEQ6AKA, seg_pos={0 0}, upload=229019]: Uploading piece... Dec 16 12:41:02.620353237 (261h12m2.491436292s) [noise=true , address=145.79.227.2:4114 , piece_id=OOHN3NOCIWV4PJQFP6PP5HNSE7AMQNIYUTNE3GUZWGZNNBIDJLVA, seg_pos={0 0}, upload=229019]: Uploading piece... Dec 16 12:41:02.620873718 (261h12m2.491956763s) [noise=true , address=77.90.31.13:4627 , piece_id=UVT3WIYHWEAODXXQEOCL44ITURWNPKQCOBJGSAOGE22FZFGW7KIA, seg_pos={0 0}, upload=229019]: Uploading piece... Dec 16 12:41:02.621508055 (261h12m2.4925911s) [noise=true , address=145.79.227.37:4565 , piece_id=LYV7QPUIRN6GBHOA3AZQZEGD3OBLV7FHJ5VQNCQNOCRR7XYJGJKA, seg_pos={0 0}, upload=229019]: Uploading piece... Dec 16 12:41:02.622063874 (261h12m2.493146929s) [noise=true , address=45.91.116.4:4939 , piece_id=33J2XYXK3HJCKKEKNYF2ZFFXVLN7SDKKPDODEKJBCB4H2SPFEI3Q, seg_pos={0 0}, upload=229019]: Uploading piece... Dec 16 12:41:02.622713400 (261h12m2.493796445s) [noise=true , address=45.91.116.8:4046 , piece_id=H2EMKIPYJ5LX4YAJYL5LL3TNANU7XC3QMZZDTSAS4VF4RKXBR45A, seg_pos={0 0}, upload=229019]: Uploading piece... Dec 16 12:41:02.623064349 (261h12m2.494147404s) [noise=true , address=145.79.227.31:4797 , piece_id=HEFSDSKFEPWBXNX6UUODOLJY6ELLCCZPWEMAEDY2YJPZHC76QU6Q, seg_pos={0 0}, upload=229019]: Uploading piece... Dec 16 12:41:02.623515127 (261h12m2.494598202s) [noise=true , address=45.91.116.5:4300 , piece_id=KR7FEGNXVBVVQNPERSU3IJBAJCKSMFSZF3K4RSQF34EFBVB2Y6TQ, seg_pos={0 0}, upload=229019]: Uploading piece... Dec 16 12:41:02.623962238 (261h12m2.495045273s) [noise=true , address=145.79.227.46:4603 , piece_id=33ZVWVMTLXVUVXONZ5XF25LBPCNRJSLW2VXNYZIAVFRT3AIX24DQ, seg_pos={0 0}, upload=229019]: Uploading piece... Dec 16 12:41:02.627737227 (261h12m2.498820272s) [noise=true , address=77.90.31.19:4844 , piece_id=PL77RE6OFQMUBS2QE2CVANQF2B4YXX2ZURW4VTOW56ONTXYVNHPA, seg_pos={0 0}, upload=229019]: Uploading piece... Dec 16 12:41:02.628421890 (261h12m2.499504925s) [noise=true , address=145.79.227.51:4411 , piece_id=2XSX4FFN7QDB42AMYAWK7RRW2ZIVUJOIO4HXRIRH5PJA4E5PIQ6A, seg_pos={0 0}, upload=229019]: Uploading piece... Dec 16 12:41:04.307455098 (261h12m4.178538143s) [upload=229019]: Next returned nil segment Dec 16 12:41:04.307483461 (261h12m4.178566506s) [upload=229019]: Waiting for error group managing segments... Dec 16 12:41:04.616131484 (261h12m4.487214529s) [noise=true , address=145.79.227.23:4545 , piece_id=HSM2EF4XUIRY5S66UYUNYCGDRYZLKXP5KNREZ637K3267KEQ6AKA, seg_pos={0 0}, upload=229019]: Done uploading piece. err: <nil> Dec 16 12:41:04.618006805 (261h12m4.48908985s) [noise=true , address=145.79.227.52:4446 , piece_id=D6C54FW7JTL54Z7K7H3EVMS7VYDZ76RNQJD3VRJXWJYVTC6CUS2A, seg_pos={0 0}, upload=229019]: Done uploading piece. err: <nil> Dec 16 12:41:04.618126533 (261h12m4.489209578s) [noise=true , address=145.79.227.42:4121 , piece_id=AMSPQNXERGDDA4FX6U3V3TQ5KGQ5P2CHFJUBT3TJOIGYXC4H7BTQ, seg_pos={0 0}, upload=229019]: Done uploading piece. err: <nil> Dec 16 12:41:04.620233455 (261h12m4.49131649s) [noise=true , address=145.79.227.37:4565 , piece_id=LYV7QPUIRN6GBHOA3AZQZEGD3OBLV7FHJ5VQNCQNOCRR7XYJGJKA, seg_pos={0 0}, upload=229019]: Done uploading piece. err: <nil> Dec 16 12:41:04.633476863 (261h12m4.504559898s) [noise=true , address=145.79.227.2:4114 , piece_id=OOHN3NOCIWV4PJQFP6PP5HNSE7AMQNIYUTNE3GUZWGZNNBIDJLVA, seg_pos={0 0}, upload=229019]: Done uploading piece. err: <nil> Dec 16 12:41:04.640074275 (261h12m4.51115732s) [noise=true , address=145.79.227.50:4374 , piece_id=NDRBX4YXQ5ZGVX2EFZNTZODGZD3F6OVBOKTIKEQFU3IMGSW6XHRQ, seg_pos={0 0}, upload=229019]: Done uploading piece. err: <nil> Dec 16 12:41:04.640631055 (261h12m4.5117141s) [noise=true , address=145.79.227.46:4603 , piece_id=33ZVWVMTLXVUVXONZ5XF25LBPCNRJSLW2VXNYZIAVFRT3AIX24DQ, seg_pos={0 0}, upload=229019]: Done uploading piece. err: <nil> Dec 16 12:41:04.654684656 (261h12m4.525767691s) [noise=true , address=145.79.227.31:4797 , piece_id=HEFSDSKFEPWBXNX6UUODOLJY6ELLCCZPWEMAEDY2YJPZHC76QU6Q, seg_pos={0 0}, upload=229019]: Done uploading piece. err: <nil> Dec 16 12:41:04.687122594 (261h12m4.558205639s) [noise=true , address=145.79.227.51:4411 , piece_id=2XSX4FFN7QDB42AMYAWK7RRW2ZIVUJOIO4HXRIRH5PJA4E5PIQ6A, seg_pos={0 0}, upload=229019]: Done uploading piece. err: <nil> Dec 16 12:41:04.699192387 (261h12m4.570275422s) [noise=true , address=145.79.227.40:4051 , piece_id=4LLC5QT5VD3Q7KTUZUNPRN7H2HS7KPAI53VKLCH2L3FSLFQJD7SA, seg_pos={0 0}, upload=229019]: Done uploading piece. err: <nil> Dec 16 12:41:04.827971536 (261h12m4.699054571s) [noise=true , address=45.91.116.8:4046 , piece_id=H2EMKIPYJ5LX4YAJYL5LL3TNANU7XC3QMZZDTSAS4VF4RKXBR45A, seg_pos={0 0}, upload=229019]: Done uploading piece. err: <nil> Dec 16 12:41:04.891714144 (261h12m4.762797189s) [noise=true , address=77.90.31.17:4773 , piece_id=ENBWT2AN2UT67GOXDGFOCYSB6MUVWFCED6HH5DG7RDJA6BOA2ZNA, seg_pos={0 0}, upload=229019]: Done uploading piece. err: <nil> Dec 16 12:41:04.897785625 (261h12m4.76886867s) [noise=true , address=77.90.31.19:4844 , piece_id=PL77RE6OFQMUBS2QE2CVANQF2B4YXX2ZURW4VTOW56ONTXYVNHPA, seg_pos={0 0}, upload=229019]: Done uploading piece. err: <nil> Dec 16 12:41:04.897858313 (261h12m4.768941348s) [seg_pos={0 0}, upload=229019]: Segment reached optimal threshold of 13 pieces. Dec 16 12:41:04.898181489 (261h12m4.769264534s) [noise=true , address=45.91.116.5:4300 , piece_id=KR7FEGNXVBVVQNPERSU3IJBAJCKSMFSZF3K4RSQF34EFBVB2Y6TQ, seg_pos={0 0}, upload=229019]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF Dec 16 12:41:04.898192810 (261h12m4.769275855s) [noise=true , address=45.91.116.5:4300 , piece_id=KR7FEGNXVBVVQNPERSU3IJBAJCKSMFSZF3K4RSQF34EFBVB2Y6TQ, seg_pos={0 0}, upload=229019]: Long tail cancellation due to optimal threshold Dec 16 12:41:04.898618681 (261h12m4.769701726s) [noise=true , address=45.91.116.4:4939 , piece_id=33J2XYXK3HJCKKEKNYF2ZFFXVLN7SDKKPDODEKJBCB4H2SPFEI3Q, seg_pos={0 0}, upload=229019]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Dec 16 12:41:04.898657345 (261h12m4.76974039s) [noise=true , address=45.91.116.4:4939 , piece_id=33J2XYXK3HJCKKEKNYF2ZFFXVLN7SDKKPDODEKJBCB4H2SPFEI3Q, seg_pos={0 0}, upload=229019]: Long tail cancellation due to optimal threshold Dec 16 12:41:04.898778977 (261h12m4.769862022s) [noise=true , address=77.90.31.13:4627 , piece_id=UVT3WIYHWEAODXXQEOCL44ITURWNPKQCOBJGSAOGE22FZFGW7KIA, seg_pos={0 0}, upload=229019]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: read tcp 188.166.225.170:41726->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 16 12:41:04.898789476 (261h12m4.769872511s) [noise=true , address=77.90.31.13:4627 , piece_id=UVT3WIYHWEAODXXQEOCL44ITURWNPKQCOBJGSAOGE22FZFGW7KIA, seg_pos={0 0}, upload=229019]: Long tail cancellation due to optimal threshold Dec 16 12:41:04.898815366 (261h12m4.769898411s) [seg_pos={0 0}, upload=229019]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 12:41:04.898903884 (261h12m4.769986939s) [upload=229019]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 12:41:04.898911098 (261h12m4.769994144s) [upload=229019]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 12:41:05.201351762 (261h12m5.072434807s) [upload=229019]: Done uploading segments... Dec 16 12:41:05.201374225 (261h12m5.07245726s) [upload=229019]: Upload finished. err: <nil> Duration: 30.003223042s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 21 out of 60 is failed (threshold=6) [.X....X.XX.......X....X...XXX......X.X...X........X.XXXXXXXX] duration: 5.001430871s trace: https://jaeger.dev.storj.io/trace/64cf1045bdaf64a5 FAIL (2025-12-16 12:40:57 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00145698s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 20 out of 60 is failed (threshold=6) [..X....X.XX.......X....X...XXX......X.X...X........X.XXXXXXX] duration: 5.002288102s trace: https://jaeger.dev.storj.io/trace/79e3a77cab1ffd0e FAIL (2025-12-16 12:40:22 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00231805s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............................XX............................] duration: 5.802793974s trace: https://jaeger.dev.storj.io/trace/61b21d2c05e36365 FAIL (2025-12-16 12:40:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 16 12:18:26.867577534 (260h49m26.73866058s) [upload=228688]: Starting upload Dec 16 12:18:26.867632178 (260h49m26.738715213s) [upload=228688]: Done starting upload Dec 16 12:18:26.867824475 (260h49m26.73890751s) [upload=228688]: Uploading segments... Dec 16 12:18:26.868036498 (260h49m26.739119533s) [upload=228688]: Got next segment. Inline: false Dec 16 12:18:26.868072116 (260h49m26.739155151s) [upload=228688]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 12:18:26.868076705 (260h49m26.73915975s) [upload=228688]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 12:18:27.112751871 (260h49m26.983834906s) [seg_pos={0 0}, upload=228688]: Begin upload segment... Dec 16 12:18:27.112835951 (260h49m26.983918996s) [seg_pos={0 0}, upload=228688]: Done begin upload segment. Dec 16 12:18:27.112906205 (260h49m26.98398925s) [noise=true , address=145.79.227.38:4978 , piece_id=3JYB42IYDMMSEFRJFAIOWPCV5JJAQOILCEGLHMSBWSXT6GPFNWSA, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.113024209 (260h49m26.984107255s) [noise=true , address=45.91.116.6:4975 , piece_id=DNIYXILY5EQJVYJSQUPYSCFJPM7UWDRWEW37ES2FGJTIVIFHQALA, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.113642677 (260h49m26.984725722s) [noise=true , address=145.79.227.41:4086 , piece_id=D2YPUZK4HQENCLATWCFHFX6FOKRDINRTPRERRUVXX63HMZNP5TVA, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.114458290 (260h49m26.985541335s) [noise=true , address=145.79.227.3:4150 , piece_id=WXZYNAKPIHDYYMWUZKBI4G7PP4EV2Y6XDQSADOJZASEBMXL7FXQA, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.114469862 (260h49m26.985552907s) [noise=true , address=45.91.116.5:4297 , piece_id=GRQJKJS4EIM7A5ATMYUSOIFYZTBBKYO72IITZLO72JDM3BKINXQQ, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.114980304 (260h49m26.986063339s) [noise=true , address=145.79.227.4:4185 , piece_id=575AA6RIXXODXK2OVQGSSMD4PWPFOXW6J6MSKIQBK33QHIONZ7XA, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.115007396 (260h49m26.986090431s) [noise=true , address=145.79.227.22:4510 , piece_id=JIZVI6JLUC24RSB2FD4AG6DFWY2W243ZDFVEBXAX6WQ77CUBYULA, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.115371549 (260h49m26.986454594s) [noise=true , address=145.79.227.2:4114 , piece_id=QIU7OSO67RMZPWXBCIVCBFVXNBFZ4MMXEUON2EW3U273W6O4XN4Q, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.115802831 (260h49m26.986885866s) [noise=true , address=77.90.31.13:4629 , piece_id=QV4CXCCUBF6COTIJRYMXMDN2U2MXBHKKFABXAM5C6AZV3XLRV7JQ, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.116457888 (260h49m26.987540923s) [noise=true , address=145.79.227.8:4331 , piece_id=CB7Q2YLIVRLUH5QIFSA3WKICUYRJQVQ5J6SRILARC72SWTMUI7LA, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.116866826 (260h49m26.987949861s) [noise=true , address=145.79.227.28:4726 , piece_id=SUMLGDF2DGXG2MXVBYRMGBW6O2V676MQFGATY6RMPAKKHVVELYHA, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.117376858 (260h49m26.988459893s) [noise=true , address=45.91.116.9:4264 , piece_id=AYTPESZCOSQFBGMYUSHEQC7YEWWZFZFAP6D4F2Y4ITRHGGSIOHAQ, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.117883373 (260h49m26.988966408s) [noise=true , address=145.79.227.20:4439 , piece_id=UMDFN6JIAD7FPWBZJKXGWCOOTHR7OHHZNUBXVG5H3Q4X2FG6FAWQ, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.118506488 (260h49m26.989589533s) [noise=true , address=77.90.31.14:4664 , piece_id=OLEOUQUB6UYPXDQDFR45ZUXDCISMBCZFEZY7CEYLGW4ISZFKLRAQ, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.119093947 (260h49m26.990176992s) [noise=true , address=77.90.31.19:4842 , piece_id=ZRYPL7FZVCKKNPRLJKV4FIONAKEK7HUDUFRIGF42TVTOHH7KLGEQ, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.120217387 (260h49m26.991300422s) [noise=true , address=77.90.31.17:4773 , piece_id=SN2X36NEBJIJEDXSLJAROMD3WGOT7DHSC6TGCI4BM7EVRUM777OQ, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:28.723929985 (260h49m28.59501302s) [upload=228688]: Next returned nil segment Dec 16 12:18:28.723956255 (260h49m28.59503929s) [upload=228688]: Waiting for error group managing segments... Dec 16 12:18:29.110968427 (260h49m28.982051472s) [noise=true , address=145.79.227.8:4331 , piece_id=CB7Q2YLIVRLUH5QIFSA3WKICUYRJQVQ5J6SRILARC72SWTMUI7LA, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.112846964 (260h49m28.983929999s) [noise=true , address=145.79.227.41:4086 , piece_id=D2YPUZK4HQENCLATWCFHFX6FOKRDINRTPRERRUVXX63HMZNP5TVA, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.115361984 (260h49m28.986445029s) [noise=true , address=145.79.227.22:4510 , piece_id=JIZVI6JLUC24RSB2FD4AG6DFWY2W243ZDFVEBXAX6WQ77CUBYULA, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.122097569 (260h49m28.993180614s) [noise=true , address=145.79.227.2:4114 , piece_id=QIU7OSO67RMZPWXBCIVCBFVXNBFZ4MMXEUON2EW3U273W6O4XN4Q, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.127395447 (260h49m28.998478482s) [noise=true , address=145.79.227.28:4726 , piece_id=SUMLGDF2DGXG2MXVBYRMGBW6O2V676MQFGATY6RMPAKKHVVELYHA, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.131068261 (260h49m29.002151306s) [noise=true , address=145.79.227.20:4439 , piece_id=UMDFN6JIAD7FPWBZJKXGWCOOTHR7OHHZNUBXVG5H3Q4X2FG6FAWQ, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.179622336 (260h49m29.050705371s) [noise=true , address=145.79.227.4:4185 , piece_id=575AA6RIXXODXK2OVQGSSMD4PWPFOXW6J6MSKIQBK33QHIONZ7XA, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.198721144 (260h49m29.069804179s) [noise=true , address=145.79.227.38:4978 , piece_id=3JYB42IYDMMSEFRJFAIOWPCV5JJAQOILCEGLHMSBWSXT6GPFNWSA, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.203940152 (260h49m29.075023197s) [noise=true , address=145.79.227.3:4150 , piece_id=WXZYNAKPIHDYYMWUZKBI4G7PP4EV2Y6XDQSADOJZASEBMXL7FXQA, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.347406548 (260h49m29.218489583s) [noise=true , address=77.90.31.17:4773 , piece_id=SN2X36NEBJIJEDXSLJAROMD3WGOT7DHSC6TGCI4BM7EVRUM777OQ, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.361791931 (260h49m29.232874966s) [noise=true , address=45.91.116.9:4264 , piece_id=AYTPESZCOSQFBGMYUSHEQC7YEWWZFZFAP6D4F2Y4ITRHGGSIOHAQ, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.447032213 (260h49m29.318115248s) [noise=true , address=45.91.116.5:4297 , piece_id=GRQJKJS4EIM7A5ATMYUSOIFYZTBBKYO72IITZLO72JDM3BKINXQQ, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.932181896 (260h49m29.803264931s) [noise=true , address=77.90.31.13:4629 , piece_id=QV4CXCCUBF6COTIJRYMXMDN2U2MXBHKKFABXAM5C6AZV3XLRV7JQ, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.932199990 (260h49m29.803283025s) [seg_pos={0 0}, upload=228688]: Segment reached optimal threshold of 13 pieces. Dec 16 12:18:29.932354895 (260h49m29.803437931s) [noise=true , address=45.91.116.6:4975 , piece_id=DNIYXILY5EQJVYJSQUPYSCFJPM7UWDRWEW37ES2FGJTIVIFHQALA, seg_pos={0 0}, upload=228688]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 16 12:18:29.932360746 (260h49m29.803443782s) [noise=true , address=45.91.116.6:4975 , piece_id=DNIYXILY5EQJVYJSQUPYSCFJPM7UWDRWEW37ES2FGJTIVIFHQALA, seg_pos={0 0}, upload=228688]: Long tail cancellation due to optimal threshold Dec 16 12:18:29.932609250 (260h49m29.803692295s) [noise=true , address=77.90.31.14:4664 , piece_id=OLEOUQUB6UYPXDQDFR45ZUXDCISMBCZFEZY7CEYLGW4ISZFKLRAQ, seg_pos={0 0}, upload=228688]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9): protocol: read tcp 188.166.225.170:44270->77.90.31.14:4664: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 16 12:18:29.932616132 (260h49m29.803699167s) [noise=true , address=77.90.31.14:4664 , piece_id=OLEOUQUB6UYPXDQDFR45ZUXDCISMBCZFEZY7CEYLGW4ISZFKLRAQ, seg_pos={0 0}, upload=228688]: Long tail cancellation due to optimal threshold Dec 16 12:18:29.932632554 (260h49m29.803715589s) [noise=true , address=77.90.31.19:4842 , piece_id=ZRYPL7FZVCKKNPRLJKV4FIONAKEK7HUDUFRIGF42TVTOHH7KLGEQ, seg_pos={0 0}, upload=228688]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Dec 16 12:18:29.932650127 (260h49m29.803733172s) [noise=true , address=77.90.31.19:4842 , piece_id=ZRYPL7FZVCKKNPRLJKV4FIONAKEK7HUDUFRIGF42TVTOHH7KLGEQ, seg_pos={0 0}, upload=228688]: Long tail cancellation due to optimal threshold Dec 16 12:18:29.932678501 (260h49m29.803761546s) [seg_pos={0 0}, upload=228688]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 12:18:29.932783451 (260h49m29.803866496s) [upload=228688]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 12:18:29.932789533 (260h49m29.803872578s) [upload=228688]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 12:18:30.222615735 (260h49m30.09369879s) [upload=228688]: Done uploading segments... Dec 16 12:18:30.222639069 (260h49m30.093722344s) [upload=228688]: Upload finished. err: <nil> Duration: 5.802820464s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............................XX...........................] duration: 13.134897788s trace: https://jaeger.dev.storj.io/trace/61b21d2c05e36365 FAIL (2025-12-16 12:39:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 16 12:18:26.867577534 (260h49m26.73866058s) [upload=228688]: Starting upload Dec 16 12:18:26.867632178 (260h49m26.738715213s) [upload=228688]: Done starting upload Dec 16 12:18:26.867824475 (260h49m26.73890751s) [upload=228688]: Uploading segments... Dec 16 12:18:26.868036498 (260h49m26.739119533s) [upload=228688]: Got next segment. Inline: false Dec 16 12:18:26.868072116 (260h49m26.739155151s) [upload=228688]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 12:18:26.868076705 (260h49m26.73915975s) [upload=228688]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 12:18:27.112751871 (260h49m26.983834906s) [seg_pos={0 0}, upload=228688]: Begin upload segment... Dec 16 12:18:27.112835951 (260h49m26.983918996s) [seg_pos={0 0}, upload=228688]: Done begin upload segment. Dec 16 12:18:27.112906205 (260h49m26.98398925s) [noise=true , address=145.79.227.38:4978 , piece_id=3JYB42IYDMMSEFRJFAIOWPCV5JJAQOILCEGLHMSBWSXT6GPFNWSA, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.113024209 (260h49m26.984107255s) [noise=true , address=45.91.116.6:4975 , piece_id=DNIYXILY5EQJVYJSQUPYSCFJPM7UWDRWEW37ES2FGJTIVIFHQALA, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.113642677 (260h49m26.984725722s) [noise=true , address=145.79.227.41:4086 , piece_id=D2YPUZK4HQENCLATWCFHFX6FOKRDINRTPRERRUVXX63HMZNP5TVA, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.114458290 (260h49m26.985541335s) [noise=true , address=145.79.227.3:4150 , piece_id=WXZYNAKPIHDYYMWUZKBI4G7PP4EV2Y6XDQSADOJZASEBMXL7FXQA, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.114469862 (260h49m26.985552907s) [noise=true , address=45.91.116.5:4297 , piece_id=GRQJKJS4EIM7A5ATMYUSOIFYZTBBKYO72IITZLO72JDM3BKINXQQ, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.114980304 (260h49m26.986063339s) [noise=true , address=145.79.227.4:4185 , piece_id=575AA6RIXXODXK2OVQGSSMD4PWPFOXW6J6MSKIQBK33QHIONZ7XA, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.115007396 (260h49m26.986090431s) [noise=true , address=145.79.227.22:4510 , piece_id=JIZVI6JLUC24RSB2FD4AG6DFWY2W243ZDFVEBXAX6WQ77CUBYULA, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.115371549 (260h49m26.986454594s) [noise=true , address=145.79.227.2:4114 , piece_id=QIU7OSO67RMZPWXBCIVCBFVXNBFZ4MMXEUON2EW3U273W6O4XN4Q, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.115802831 (260h49m26.986885866s) [noise=true , address=77.90.31.13:4629 , piece_id=QV4CXCCUBF6COTIJRYMXMDN2U2MXBHKKFABXAM5C6AZV3XLRV7JQ, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.116457888 (260h49m26.987540923s) [noise=true , address=145.79.227.8:4331 , piece_id=CB7Q2YLIVRLUH5QIFSA3WKICUYRJQVQ5J6SRILARC72SWTMUI7LA, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.116866826 (260h49m26.987949861s) [noise=true , address=145.79.227.28:4726 , piece_id=SUMLGDF2DGXG2MXVBYRMGBW6O2V676MQFGATY6RMPAKKHVVELYHA, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.117376858 (260h49m26.988459893s) [noise=true , address=45.91.116.9:4264 , piece_id=AYTPESZCOSQFBGMYUSHEQC7YEWWZFZFAP6D4F2Y4ITRHGGSIOHAQ, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.117883373 (260h49m26.988966408s) [noise=true , address=145.79.227.20:4439 , piece_id=UMDFN6JIAD7FPWBZJKXGWCOOTHR7OHHZNUBXVG5H3Q4X2FG6FAWQ, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.118506488 (260h49m26.989589533s) [noise=true , address=77.90.31.14:4664 , piece_id=OLEOUQUB6UYPXDQDFR45ZUXDCISMBCZFEZY7CEYLGW4ISZFKLRAQ, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.119093947 (260h49m26.990176992s) [noise=true , address=77.90.31.19:4842 , piece_id=ZRYPL7FZVCKKNPRLJKV4FIONAKEK7HUDUFRIGF42TVTOHH7KLGEQ, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.120217387 (260h49m26.991300422s) [noise=true , address=77.90.31.17:4773 , piece_id=SN2X36NEBJIJEDXSLJAROMD3WGOT7DHSC6TGCI4BM7EVRUM777OQ, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:28.723929985 (260h49m28.59501302s) [upload=228688]: Next returned nil segment Dec 16 12:18:28.723956255 (260h49m28.59503929s) [upload=228688]: Waiting for error group managing segments... Dec 16 12:18:29.110968427 (260h49m28.982051472s) [noise=true , address=145.79.227.8:4331 , piece_id=CB7Q2YLIVRLUH5QIFSA3WKICUYRJQVQ5J6SRILARC72SWTMUI7LA, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.112846964 (260h49m28.983929999s) [noise=true , address=145.79.227.41:4086 , piece_id=D2YPUZK4HQENCLATWCFHFX6FOKRDINRTPRERRUVXX63HMZNP5TVA, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.115361984 (260h49m28.986445029s) [noise=true , address=145.79.227.22:4510 , piece_id=JIZVI6JLUC24RSB2FD4AG6DFWY2W243ZDFVEBXAX6WQ77CUBYULA, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.122097569 (260h49m28.993180614s) [noise=true , address=145.79.227.2:4114 , piece_id=QIU7OSO67RMZPWXBCIVCBFVXNBFZ4MMXEUON2EW3U273W6O4XN4Q, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.127395447 (260h49m28.998478482s) [noise=true , address=145.79.227.28:4726 , piece_id=SUMLGDF2DGXG2MXVBYRMGBW6O2V676MQFGATY6RMPAKKHVVELYHA, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.131068261 (260h49m29.002151306s) [noise=true , address=145.79.227.20:4439 , piece_id=UMDFN6JIAD7FPWBZJKXGWCOOTHR7OHHZNUBXVG5H3Q4X2FG6FAWQ, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.179622336 (260h49m29.050705371s) [noise=true , address=145.79.227.4:4185 , piece_id=575AA6RIXXODXK2OVQGSSMD4PWPFOXW6J6MSKIQBK33QHIONZ7XA, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.198721144 (260h49m29.069804179s) [noise=true , address=145.79.227.38:4978 , piece_id=3JYB42IYDMMSEFRJFAIOWPCV5JJAQOILCEGLHMSBWSXT6GPFNWSA, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.203940152 (260h49m29.075023197s) [noise=true , address=145.79.227.3:4150 , piece_id=WXZYNAKPIHDYYMWUZKBI4G7PP4EV2Y6XDQSADOJZASEBMXL7FXQA, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.347406548 (260h49m29.218489583s) [noise=true , address=77.90.31.17:4773 , piece_id=SN2X36NEBJIJEDXSLJAROMD3WGOT7DHSC6TGCI4BM7EVRUM777OQ, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.361791931 (260h49m29.232874966s) [noise=true , address=45.91.116.9:4264 , piece_id=AYTPESZCOSQFBGMYUSHEQC7YEWWZFZFAP6D4F2Y4ITRHGGSIOHAQ, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.447032213 (260h49m29.318115248s) [noise=true , address=45.91.116.5:4297 , piece_id=GRQJKJS4EIM7A5ATMYUSOIFYZTBBKYO72IITZLO72JDM3BKINXQQ, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.932181896 (260h49m29.803264931s) [noise=true , address=77.90.31.13:4629 , piece_id=QV4CXCCUBF6COTIJRYMXMDN2U2MXBHKKFABXAM5C6AZV3XLRV7JQ, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.932199990 (260h49m29.803283025s) [seg_pos={0 0}, upload=228688]: Segment reached optimal threshold of 13 pieces. Dec 16 12:18:29.932354895 (260h49m29.803437931s) [noise=true , address=45.91.116.6:4975 , piece_id=DNIYXILY5EQJVYJSQUPYSCFJPM7UWDRWEW37ES2FGJTIVIFHQALA, seg_pos={0 0}, upload=228688]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 16 12:18:29.932360746 (260h49m29.803443782s) [noise=true , address=45.91.116.6:4975 , piece_id=DNIYXILY5EQJVYJSQUPYSCFJPM7UWDRWEW37ES2FGJTIVIFHQALA, seg_pos={0 0}, upload=228688]: Long tail cancellation due to optimal threshold Dec 16 12:18:29.932609250 (260h49m29.803692295s) [noise=true , address=77.90.31.14:4664 , piece_id=OLEOUQUB6UYPXDQDFR45ZUXDCISMBCZFEZY7CEYLGW4ISZFKLRAQ, seg_pos={0 0}, upload=228688]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9): protocol: read tcp 188.166.225.170:44270->77.90.31.14:4664: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 16 12:18:29.932616132 (260h49m29.803699167s) [noise=true , address=77.90.31.14:4664 , piece_id=OLEOUQUB6UYPXDQDFR45ZUXDCISMBCZFEZY7CEYLGW4ISZFKLRAQ, seg_pos={0 0}, upload=228688]: Long tail cancellation due to optimal threshold Dec 16 12:18:29.932632554 (260h49m29.803715589s) [noise=true , address=77.90.31.19:4842 , piece_id=ZRYPL7FZVCKKNPRLJKV4FIONAKEK7HUDUFRIGF42TVTOHH7KLGEQ, seg_pos={0 0}, upload=228688]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Dec 16 12:18:29.932650127 (260h49m29.803733172s) [noise=true , address=77.90.31.19:4842 , piece_id=ZRYPL7FZVCKKNPRLJKV4FIONAKEK7HUDUFRIGF42TVTOHH7KLGEQ, seg_pos={0 0}, upload=228688]: Long tail cancellation due to optimal threshold Dec 16 12:18:29.932678501 (260h49m29.803761546s) [seg_pos={0 0}, upload=228688]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 12:18:29.932783451 (260h49m29.803866496s) [upload=228688]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 12:18:29.932789533 (260h49m29.803872578s) [upload=228688]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 12:18:30.222615735 (260h49m30.09369879s) [upload=228688]: Done uploading segments... Dec 16 12:18:30.222639069 (260h49m30.093722344s) [upload=228688]: Upload finished. err: <nil> Duration: 13.134927645s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 19 out of 60 is failed (threshold=6) [...X....X.XX.......X....X...XXX......X.X...X........X.XXXXXX] duration: 5.001507538s trace: https://jaeger.dev.storj.io/trace/351f6c315b583320 FAIL (2025-12-16 12:39:33 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001528129s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................................XX..........................] duration: 5.847465096s trace: https://jaeger.dev.storj.io/trace/61b21d2c05e36365 FAIL (2025-12-16 12:38:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 16 12:18:26.867577534 (260h49m26.73866058s) [upload=228688]: Starting upload Dec 16 12:18:26.867632178 (260h49m26.738715213s) [upload=228688]: Done starting upload Dec 16 12:18:26.867824475 (260h49m26.73890751s) [upload=228688]: Uploading segments... Dec 16 12:18:26.868036498 (260h49m26.739119533s) [upload=228688]: Got next segment. Inline: false Dec 16 12:18:26.868072116 (260h49m26.739155151s) [upload=228688]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 12:18:26.868076705 (260h49m26.73915975s) [upload=228688]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 12:18:27.112751871 (260h49m26.983834906s) [seg_pos={0 0}, upload=228688]: Begin upload segment... Dec 16 12:18:27.112835951 (260h49m26.983918996s) [seg_pos={0 0}, upload=228688]: Done begin upload segment. Dec 16 12:18:27.112906205 (260h49m26.98398925s) [noise=true , address=145.79.227.38:4978 , piece_id=3JYB42IYDMMSEFRJFAIOWPCV5JJAQOILCEGLHMSBWSXT6GPFNWSA, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.113024209 (260h49m26.984107255s) [noise=true , address=45.91.116.6:4975 , piece_id=DNIYXILY5EQJVYJSQUPYSCFJPM7UWDRWEW37ES2FGJTIVIFHQALA, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.113642677 (260h49m26.984725722s) [noise=true , address=145.79.227.41:4086 , piece_id=D2YPUZK4HQENCLATWCFHFX6FOKRDINRTPRERRUVXX63HMZNP5TVA, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.114458290 (260h49m26.985541335s) [noise=true , address=145.79.227.3:4150 , piece_id=WXZYNAKPIHDYYMWUZKBI4G7PP4EV2Y6XDQSADOJZASEBMXL7FXQA, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.114469862 (260h49m26.985552907s) [noise=true , address=45.91.116.5:4297 , piece_id=GRQJKJS4EIM7A5ATMYUSOIFYZTBBKYO72IITZLO72JDM3BKINXQQ, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.114980304 (260h49m26.986063339s) [noise=true , address=145.79.227.4:4185 , piece_id=575AA6RIXXODXK2OVQGSSMD4PWPFOXW6J6MSKIQBK33QHIONZ7XA, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.115007396 (260h49m26.986090431s) [noise=true , address=145.79.227.22:4510 , piece_id=JIZVI6JLUC24RSB2FD4AG6DFWY2W243ZDFVEBXAX6WQ77CUBYULA, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.115371549 (260h49m26.986454594s) [noise=true , address=145.79.227.2:4114 , piece_id=QIU7OSO67RMZPWXBCIVCBFVXNBFZ4MMXEUON2EW3U273W6O4XN4Q, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.115802831 (260h49m26.986885866s) [noise=true , address=77.90.31.13:4629 , piece_id=QV4CXCCUBF6COTIJRYMXMDN2U2MXBHKKFABXAM5C6AZV3XLRV7JQ, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.116457888 (260h49m26.987540923s) [noise=true , address=145.79.227.8:4331 , piece_id=CB7Q2YLIVRLUH5QIFSA3WKICUYRJQVQ5J6SRILARC72SWTMUI7LA, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.116866826 (260h49m26.987949861s) [noise=true , address=145.79.227.28:4726 , piece_id=SUMLGDF2DGXG2MXVBYRMGBW6O2V676MQFGATY6RMPAKKHVVELYHA, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.117376858 (260h49m26.988459893s) [noise=true , address=45.91.116.9:4264 , piece_id=AYTPESZCOSQFBGMYUSHEQC7YEWWZFZFAP6D4F2Y4ITRHGGSIOHAQ, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.117883373 (260h49m26.988966408s) [noise=true , address=145.79.227.20:4439 , piece_id=UMDFN6JIAD7FPWBZJKXGWCOOTHR7OHHZNUBXVG5H3Q4X2FG6FAWQ, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.118506488 (260h49m26.989589533s) [noise=true , address=77.90.31.14:4664 , piece_id=OLEOUQUB6UYPXDQDFR45ZUXDCISMBCZFEZY7CEYLGW4ISZFKLRAQ, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.119093947 (260h49m26.990176992s) [noise=true , address=77.90.31.19:4842 , piece_id=ZRYPL7FZVCKKNPRLJKV4FIONAKEK7HUDUFRIGF42TVTOHH7KLGEQ, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.120217387 (260h49m26.991300422s) [noise=true , address=77.90.31.17:4773 , piece_id=SN2X36NEBJIJEDXSLJAROMD3WGOT7DHSC6TGCI4BM7EVRUM777OQ, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:28.723929985 (260h49m28.59501302s) [upload=228688]: Next returned nil segment Dec 16 12:18:28.723956255 (260h49m28.59503929s) [upload=228688]: Waiting for error group managing segments... Dec 16 12:18:29.110968427 (260h49m28.982051472s) [noise=true , address=145.79.227.8:4331 , piece_id=CB7Q2YLIVRLUH5QIFSA3WKICUYRJQVQ5J6SRILARC72SWTMUI7LA, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.112846964 (260h49m28.983929999s) [noise=true , address=145.79.227.41:4086 , piece_id=D2YPUZK4HQENCLATWCFHFX6FOKRDINRTPRERRUVXX63HMZNP5TVA, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.115361984 (260h49m28.986445029s) [noise=true , address=145.79.227.22:4510 , piece_id=JIZVI6JLUC24RSB2FD4AG6DFWY2W243ZDFVEBXAX6WQ77CUBYULA, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.122097569 (260h49m28.993180614s) [noise=true , address=145.79.227.2:4114 , piece_id=QIU7OSO67RMZPWXBCIVCBFVXNBFZ4MMXEUON2EW3U273W6O4XN4Q, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.127395447 (260h49m28.998478482s) [noise=true , address=145.79.227.28:4726 , piece_id=SUMLGDF2DGXG2MXVBYRMGBW6O2V676MQFGATY6RMPAKKHVVELYHA, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.131068261 (260h49m29.002151306s) [noise=true , address=145.79.227.20:4439 , piece_id=UMDFN6JIAD7FPWBZJKXGWCOOTHR7OHHZNUBXVG5H3Q4X2FG6FAWQ, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.179622336 (260h49m29.050705371s) [noise=true , address=145.79.227.4:4185 , piece_id=575AA6RIXXODXK2OVQGSSMD4PWPFOXW6J6MSKIQBK33QHIONZ7XA, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.198721144 (260h49m29.069804179s) [noise=true , address=145.79.227.38:4978 , piece_id=3JYB42IYDMMSEFRJFAIOWPCV5JJAQOILCEGLHMSBWSXT6GPFNWSA, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.203940152 (260h49m29.075023197s) [noise=true , address=145.79.227.3:4150 , piece_id=WXZYNAKPIHDYYMWUZKBI4G7PP4EV2Y6XDQSADOJZASEBMXL7FXQA, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.347406548 (260h49m29.218489583s) [noise=true , address=77.90.31.17:4773 , piece_id=SN2X36NEBJIJEDXSLJAROMD3WGOT7DHSC6TGCI4BM7EVRUM777OQ, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.361791931 (260h49m29.232874966s) [noise=true , address=45.91.116.9:4264 , piece_id=AYTPESZCOSQFBGMYUSHEQC7YEWWZFZFAP6D4F2Y4ITRHGGSIOHAQ, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.447032213 (260h49m29.318115248s) [noise=true , address=45.91.116.5:4297 , piece_id=GRQJKJS4EIM7A5ATMYUSOIFYZTBBKYO72IITZLO72JDM3BKINXQQ, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.932181896 (260h49m29.803264931s) [noise=true , address=77.90.31.13:4629 , piece_id=QV4CXCCUBF6COTIJRYMXMDN2U2MXBHKKFABXAM5C6AZV3XLRV7JQ, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.932199990 (260h49m29.803283025s) [seg_pos={0 0}, upload=228688]: Segment reached optimal threshold of 13 pieces. Dec 16 12:18:29.932354895 (260h49m29.803437931s) [noise=true , address=45.91.116.6:4975 , piece_id=DNIYXILY5EQJVYJSQUPYSCFJPM7UWDRWEW37ES2FGJTIVIFHQALA, seg_pos={0 0}, upload=228688]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 16 12:18:29.932360746 (260h49m29.803443782s) [noise=true , address=45.91.116.6:4975 , piece_id=DNIYXILY5EQJVYJSQUPYSCFJPM7UWDRWEW37ES2FGJTIVIFHQALA, seg_pos={0 0}, upload=228688]: Long tail cancellation due to optimal threshold Dec 16 12:18:29.932609250 (260h49m29.803692295s) [noise=true , address=77.90.31.14:4664 , piece_id=OLEOUQUB6UYPXDQDFR45ZUXDCISMBCZFEZY7CEYLGW4ISZFKLRAQ, seg_pos={0 0}, upload=228688]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9): protocol: read tcp 188.166.225.170:44270->77.90.31.14:4664: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 16 12:18:29.932616132 (260h49m29.803699167s) [noise=true , address=77.90.31.14:4664 , piece_id=OLEOUQUB6UYPXDQDFR45ZUXDCISMBCZFEZY7CEYLGW4ISZFKLRAQ, seg_pos={0 0}, upload=228688]: Long tail cancellation due to optimal threshold Dec 16 12:18:29.932632554 (260h49m29.803715589s) [noise=true , address=77.90.31.19:4842 , piece_id=ZRYPL7FZVCKKNPRLJKV4FIONAKEK7HUDUFRIGF42TVTOHH7KLGEQ, seg_pos={0 0}, upload=228688]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Dec 16 12:18:29.932650127 (260h49m29.803733172s) [noise=true , address=77.90.31.19:4842 , piece_id=ZRYPL7FZVCKKNPRLJKV4FIONAKEK7HUDUFRIGF42TVTOHH7KLGEQ, seg_pos={0 0}, upload=228688]: Long tail cancellation due to optimal threshold Dec 16 12:18:29.932678501 (260h49m29.803761546s) [seg_pos={0 0}, upload=228688]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 12:18:29.932783451 (260h49m29.803866496s) [upload=228688]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 12:18:29.932789533 (260h49m29.803872578s) [upload=228688]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 12:18:30.222615735 (260h49m30.09369879s) [upload=228688]: Done uploading segments... Dec 16 12:18:30.222639069 (260h49m30.093722344s) [upload=228688]: Upload finished. err: <nil> Duration: 5.847492168s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 18 out of 60 is failed (threshold=6) [....X....X.XX.......X....X...XXX......X.X...X........X.XXXXX] duration: 5.000855596s trace: https://jaeger.dev.storj.io/trace/97d817b503bae97 FAIL (2025-12-16 12:38:36 UTC) orbiter warning: context deadline exceeded Duration: 5.000875424s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 17 out of 60 is failed (threshold=6) [.....X....X.XX.......X....X...XXX......X.X...X........X.XXXX] duration: 5.001031714s trace: https://jaeger.dev.storj.io/trace/7647b76b8121b2a0 FAIL (2025-12-16 12:37:52 UTC) orbiter warning: context deadline exceeded Duration: 5.001058834s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................................XX.........................] duration: 5.956961917s trace: https://jaeger.dev.storj.io/trace/61b21d2c05e36365 FAIL (2025-12-16 12:37:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 16 12:18:26.867577534 (260h49m26.73866058s) [upload=228688]: Starting upload Dec 16 12:18:26.867632178 (260h49m26.738715213s) [upload=228688]: Done starting upload Dec 16 12:18:26.867824475 (260h49m26.73890751s) [upload=228688]: Uploading segments... Dec 16 12:18:26.868036498 (260h49m26.739119533s) [upload=228688]: Got next segment. Inline: false Dec 16 12:18:26.868072116 (260h49m26.739155151s) [upload=228688]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 12:18:26.868076705 (260h49m26.73915975s) [upload=228688]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 12:18:27.112751871 (260h49m26.983834906s) [seg_pos={0 0}, upload=228688]: Begin upload segment... Dec 16 12:18:27.112835951 (260h49m26.983918996s) [seg_pos={0 0}, upload=228688]: Done begin upload segment. Dec 16 12:18:27.112906205 (260h49m26.98398925s) [noise=true , address=145.79.227.38:4978 , piece_id=3JYB42IYDMMSEFRJFAIOWPCV5JJAQOILCEGLHMSBWSXT6GPFNWSA, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.113024209 (260h49m26.984107255s) [noise=true , address=45.91.116.6:4975 , piece_id=DNIYXILY5EQJVYJSQUPYSCFJPM7UWDRWEW37ES2FGJTIVIFHQALA, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.113642677 (260h49m26.984725722s) [noise=true , address=145.79.227.41:4086 , piece_id=D2YPUZK4HQENCLATWCFHFX6FOKRDINRTPRERRUVXX63HMZNP5TVA, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.114458290 (260h49m26.985541335s) [noise=true , address=145.79.227.3:4150 , piece_id=WXZYNAKPIHDYYMWUZKBI4G7PP4EV2Y6XDQSADOJZASEBMXL7FXQA, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.114469862 (260h49m26.985552907s) [noise=true , address=45.91.116.5:4297 , piece_id=GRQJKJS4EIM7A5ATMYUSOIFYZTBBKYO72IITZLO72JDM3BKINXQQ, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.114980304 (260h49m26.986063339s) [noise=true , address=145.79.227.4:4185 , piece_id=575AA6RIXXODXK2OVQGSSMD4PWPFOXW6J6MSKIQBK33QHIONZ7XA, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.115007396 (260h49m26.986090431s) [noise=true , address=145.79.227.22:4510 , piece_id=JIZVI6JLUC24RSB2FD4AG6DFWY2W243ZDFVEBXAX6WQ77CUBYULA, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.115371549 (260h49m26.986454594s) [noise=true , address=145.79.227.2:4114 , piece_id=QIU7OSO67RMZPWXBCIVCBFVXNBFZ4MMXEUON2EW3U273W6O4XN4Q, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.115802831 (260h49m26.986885866s) [noise=true , address=77.90.31.13:4629 , piece_id=QV4CXCCUBF6COTIJRYMXMDN2U2MXBHKKFABXAM5C6AZV3XLRV7JQ, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.116457888 (260h49m26.987540923s) [noise=true , address=145.79.227.8:4331 , piece_id=CB7Q2YLIVRLUH5QIFSA3WKICUYRJQVQ5J6SRILARC72SWTMUI7LA, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.116866826 (260h49m26.987949861s) [noise=true , address=145.79.227.28:4726 , piece_id=SUMLGDF2DGXG2MXVBYRMGBW6O2V676MQFGATY6RMPAKKHVVELYHA, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.117376858 (260h49m26.988459893s) [noise=true , address=45.91.116.9:4264 , piece_id=AYTPESZCOSQFBGMYUSHEQC7YEWWZFZFAP6D4F2Y4ITRHGGSIOHAQ, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.117883373 (260h49m26.988966408s) [noise=true , address=145.79.227.20:4439 , piece_id=UMDFN6JIAD7FPWBZJKXGWCOOTHR7OHHZNUBXVG5H3Q4X2FG6FAWQ, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.118506488 (260h49m26.989589533s) [noise=true , address=77.90.31.14:4664 , piece_id=OLEOUQUB6UYPXDQDFR45ZUXDCISMBCZFEZY7CEYLGW4ISZFKLRAQ, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.119093947 (260h49m26.990176992s) [noise=true , address=77.90.31.19:4842 , piece_id=ZRYPL7FZVCKKNPRLJKV4FIONAKEK7HUDUFRIGF42TVTOHH7KLGEQ, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.120217387 (260h49m26.991300422s) [noise=true , address=77.90.31.17:4773 , piece_id=SN2X36NEBJIJEDXSLJAROMD3WGOT7DHSC6TGCI4BM7EVRUM777OQ, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:28.723929985 (260h49m28.59501302s) [upload=228688]: Next returned nil segment Dec 16 12:18:28.723956255 (260h49m28.59503929s) [upload=228688]: Waiting for error group managing segments... Dec 16 12:18:29.110968427 (260h49m28.982051472s) [noise=true , address=145.79.227.8:4331 , piece_id=CB7Q2YLIVRLUH5QIFSA3WKICUYRJQVQ5J6SRILARC72SWTMUI7LA, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.112846964 (260h49m28.983929999s) [noise=true , address=145.79.227.41:4086 , piece_id=D2YPUZK4HQENCLATWCFHFX6FOKRDINRTPRERRUVXX63HMZNP5TVA, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.115361984 (260h49m28.986445029s) [noise=true , address=145.79.227.22:4510 , piece_id=JIZVI6JLUC24RSB2FD4AG6DFWY2W243ZDFVEBXAX6WQ77CUBYULA, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.122097569 (260h49m28.993180614s) [noise=true , address=145.79.227.2:4114 , piece_id=QIU7OSO67RMZPWXBCIVCBFVXNBFZ4MMXEUON2EW3U273W6O4XN4Q, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.127395447 (260h49m28.998478482s) [noise=true , address=145.79.227.28:4726 , piece_id=SUMLGDF2DGXG2MXVBYRMGBW6O2V676MQFGATY6RMPAKKHVVELYHA, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.131068261 (260h49m29.002151306s) [noise=true , address=145.79.227.20:4439 , piece_id=UMDFN6JIAD7FPWBZJKXGWCOOTHR7OHHZNUBXVG5H3Q4X2FG6FAWQ, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.179622336 (260h49m29.050705371s) [noise=true , address=145.79.227.4:4185 , piece_id=575AA6RIXXODXK2OVQGSSMD4PWPFOXW6J6MSKIQBK33QHIONZ7XA, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.198721144 (260h49m29.069804179s) [noise=true , address=145.79.227.38:4978 , piece_id=3JYB42IYDMMSEFRJFAIOWPCV5JJAQOILCEGLHMSBWSXT6GPFNWSA, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.203940152 (260h49m29.075023197s) [noise=true , address=145.79.227.3:4150 , piece_id=WXZYNAKPIHDYYMWUZKBI4G7PP4EV2Y6XDQSADOJZASEBMXL7FXQA, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.347406548 (260h49m29.218489583s) [noise=true , address=77.90.31.17:4773 , piece_id=SN2X36NEBJIJEDXSLJAROMD3WGOT7DHSC6TGCI4BM7EVRUM777OQ, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.361791931 (260h49m29.232874966s) [noise=true , address=45.91.116.9:4264 , piece_id=AYTPESZCOSQFBGMYUSHEQC7YEWWZFZFAP6D4F2Y4ITRHGGSIOHAQ, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.447032213 (260h49m29.318115248s) [noise=true , address=45.91.116.5:4297 , piece_id=GRQJKJS4EIM7A5ATMYUSOIFYZTBBKYO72IITZLO72JDM3BKINXQQ, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.932181896 (260h49m29.803264931s) [noise=true , address=77.90.31.13:4629 , piece_id=QV4CXCCUBF6COTIJRYMXMDN2U2MXBHKKFABXAM5C6AZV3XLRV7JQ, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.932199990 (260h49m29.803283025s) [seg_pos={0 0}, upload=228688]: Segment reached optimal threshold of 13 pieces. Dec 16 12:18:29.932354895 (260h49m29.803437931s) [noise=true , address=45.91.116.6:4975 , piece_id=DNIYXILY5EQJVYJSQUPYSCFJPM7UWDRWEW37ES2FGJTIVIFHQALA, seg_pos={0 0}, upload=228688]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 16 12:18:29.932360746 (260h49m29.803443782s) [noise=true , address=45.91.116.6:4975 , piece_id=DNIYXILY5EQJVYJSQUPYSCFJPM7UWDRWEW37ES2FGJTIVIFHQALA, seg_pos={0 0}, upload=228688]: Long tail cancellation due to optimal threshold Dec 16 12:18:29.932609250 (260h49m29.803692295s) [noise=true , address=77.90.31.14:4664 , piece_id=OLEOUQUB6UYPXDQDFR45ZUXDCISMBCZFEZY7CEYLGW4ISZFKLRAQ, seg_pos={0 0}, upload=228688]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9): protocol: read tcp 188.166.225.170:44270->77.90.31.14:4664: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 16 12:18:29.932616132 (260h49m29.803699167s) [noise=true , address=77.90.31.14:4664 , piece_id=OLEOUQUB6UYPXDQDFR45ZUXDCISMBCZFEZY7CEYLGW4ISZFKLRAQ, seg_pos={0 0}, upload=228688]: Long tail cancellation due to optimal threshold Dec 16 12:18:29.932632554 (260h49m29.803715589s) [noise=true , address=77.90.31.19:4842 , piece_id=ZRYPL7FZVCKKNPRLJKV4FIONAKEK7HUDUFRIGF42TVTOHH7KLGEQ, seg_pos={0 0}, upload=228688]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Dec 16 12:18:29.932650127 (260h49m29.803733172s) [noise=true , address=77.90.31.19:4842 , piece_id=ZRYPL7FZVCKKNPRLJKV4FIONAKEK7HUDUFRIGF42TVTOHH7KLGEQ, seg_pos={0 0}, upload=228688]: Long tail cancellation due to optimal threshold Dec 16 12:18:29.932678501 (260h49m29.803761546s) [seg_pos={0 0}, upload=228688]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 12:18:29.932783451 (260h49m29.803866496s) [upload=228688]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 12:18:29.932789533 (260h49m29.803872578s) [upload=228688]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 12:18:30.222615735 (260h49m30.09369879s) [upload=228688]: Done uploading segments... Dec 16 12:18:30.222639069 (260h49m30.093722344s) [upload=228688]: Upload finished. err: <nil> Duration: 5.956981324s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 16 out of 60 is failed (threshold=6) [......X....X.XX.......X....X...XXX......X.X...X........X.XXX] duration: 5.001311506s trace: https://jaeger.dev.storj.io/trace/180708ea4737778d FAIL (2025-12-16 12:36:54 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001365028s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 15 out of 60 is failed (threshold=6) [.......X....X.XX.......X....X...XXX......X.X...X........X.XX] duration: 5.000759193s trace: https://jaeger.dev.storj.io/trace/610a8f32c44cea1 FAIL (2025-12-16 12:36:29 UTC) orbiter warning: context deadline exceeded Duration: 5.000782447s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................................XX........................] duration: 5.632237653s trace: https://jaeger.dev.storj.io/trace/61b21d2c05e36365 FAIL (2025-12-16 12:36:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 16 12:18:26.867577534 (260h49m26.73866058s) [upload=228688]: Starting upload Dec 16 12:18:26.867632178 (260h49m26.738715213s) [upload=228688]: Done starting upload Dec 16 12:18:26.867824475 (260h49m26.73890751s) [upload=228688]: Uploading segments... Dec 16 12:18:26.868036498 (260h49m26.739119533s) [upload=228688]: Got next segment. Inline: false Dec 16 12:18:26.868072116 (260h49m26.739155151s) [upload=228688]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 12:18:26.868076705 (260h49m26.73915975s) [upload=228688]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 12:18:27.112751871 (260h49m26.983834906s) [seg_pos={0 0}, upload=228688]: Begin upload segment... Dec 16 12:18:27.112835951 (260h49m26.983918996s) [seg_pos={0 0}, upload=228688]: Done begin upload segment. Dec 16 12:18:27.112906205 (260h49m26.98398925s) [noise=true , address=145.79.227.38:4978 , piece_id=3JYB42IYDMMSEFRJFAIOWPCV5JJAQOILCEGLHMSBWSXT6GPFNWSA, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.113024209 (260h49m26.984107255s) [noise=true , address=45.91.116.6:4975 , piece_id=DNIYXILY5EQJVYJSQUPYSCFJPM7UWDRWEW37ES2FGJTIVIFHQALA, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.113642677 (260h49m26.984725722s) [noise=true , address=145.79.227.41:4086 , piece_id=D2YPUZK4HQENCLATWCFHFX6FOKRDINRTPRERRUVXX63HMZNP5TVA, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.114458290 (260h49m26.985541335s) [noise=true , address=145.79.227.3:4150 , piece_id=WXZYNAKPIHDYYMWUZKBI4G7PP4EV2Y6XDQSADOJZASEBMXL7FXQA, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.114469862 (260h49m26.985552907s) [noise=true , address=45.91.116.5:4297 , piece_id=GRQJKJS4EIM7A5ATMYUSOIFYZTBBKYO72IITZLO72JDM3BKINXQQ, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.114980304 (260h49m26.986063339s) [noise=true , address=145.79.227.4:4185 , piece_id=575AA6RIXXODXK2OVQGSSMD4PWPFOXW6J6MSKIQBK33QHIONZ7XA, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.115007396 (260h49m26.986090431s) [noise=true , address=145.79.227.22:4510 , piece_id=JIZVI6JLUC24RSB2FD4AG6DFWY2W243ZDFVEBXAX6WQ77CUBYULA, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.115371549 (260h49m26.986454594s) [noise=true , address=145.79.227.2:4114 , piece_id=QIU7OSO67RMZPWXBCIVCBFVXNBFZ4MMXEUON2EW3U273W6O4XN4Q, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.115802831 (260h49m26.986885866s) [noise=true , address=77.90.31.13:4629 , piece_id=QV4CXCCUBF6COTIJRYMXMDN2U2MXBHKKFABXAM5C6AZV3XLRV7JQ, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.116457888 (260h49m26.987540923s) [noise=true , address=145.79.227.8:4331 , piece_id=CB7Q2YLIVRLUH5QIFSA3WKICUYRJQVQ5J6SRILARC72SWTMUI7LA, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.116866826 (260h49m26.987949861s) [noise=true , address=145.79.227.28:4726 , piece_id=SUMLGDF2DGXG2MXVBYRMGBW6O2V676MQFGATY6RMPAKKHVVELYHA, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.117376858 (260h49m26.988459893s) [noise=true , address=45.91.116.9:4264 , piece_id=AYTPESZCOSQFBGMYUSHEQC7YEWWZFZFAP6D4F2Y4ITRHGGSIOHAQ, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.117883373 (260h49m26.988966408s) [noise=true , address=145.79.227.20:4439 , piece_id=UMDFN6JIAD7FPWBZJKXGWCOOTHR7OHHZNUBXVG5H3Q4X2FG6FAWQ, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.118506488 (260h49m26.989589533s) [noise=true , address=77.90.31.14:4664 , piece_id=OLEOUQUB6UYPXDQDFR45ZUXDCISMBCZFEZY7CEYLGW4ISZFKLRAQ, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.119093947 (260h49m26.990176992s) [noise=true , address=77.90.31.19:4842 , piece_id=ZRYPL7FZVCKKNPRLJKV4FIONAKEK7HUDUFRIGF42TVTOHH7KLGEQ, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.120217387 (260h49m26.991300422s) [noise=true , address=77.90.31.17:4773 , piece_id=SN2X36NEBJIJEDXSLJAROMD3WGOT7DHSC6TGCI4BM7EVRUM777OQ, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:28.723929985 (260h49m28.59501302s) [upload=228688]: Next returned nil segment Dec 16 12:18:28.723956255 (260h49m28.59503929s) [upload=228688]: Waiting for error group managing segments... Dec 16 12:18:29.110968427 (260h49m28.982051472s) [noise=true , address=145.79.227.8:4331 , piece_id=CB7Q2YLIVRLUH5QIFSA3WKICUYRJQVQ5J6SRILARC72SWTMUI7LA, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.112846964 (260h49m28.983929999s) [noise=true , address=145.79.227.41:4086 , piece_id=D2YPUZK4HQENCLATWCFHFX6FOKRDINRTPRERRUVXX63HMZNP5TVA, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.115361984 (260h49m28.986445029s) [noise=true , address=145.79.227.22:4510 , piece_id=JIZVI6JLUC24RSB2FD4AG6DFWY2W243ZDFVEBXAX6WQ77CUBYULA, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.122097569 (260h49m28.993180614s) [noise=true , address=145.79.227.2:4114 , piece_id=QIU7OSO67RMZPWXBCIVCBFVXNBFZ4MMXEUON2EW3U273W6O4XN4Q, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.127395447 (260h49m28.998478482s) [noise=true , address=145.79.227.28:4726 , piece_id=SUMLGDF2DGXG2MXVBYRMGBW6O2V676MQFGATY6RMPAKKHVVELYHA, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.131068261 (260h49m29.002151306s) [noise=true , address=145.79.227.20:4439 , piece_id=UMDFN6JIAD7FPWBZJKXGWCOOTHR7OHHZNUBXVG5H3Q4X2FG6FAWQ, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.179622336 (260h49m29.050705371s) [noise=true , address=145.79.227.4:4185 , piece_id=575AA6RIXXODXK2OVQGSSMD4PWPFOXW6J6MSKIQBK33QHIONZ7XA, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.198721144 (260h49m29.069804179s) [noise=true , address=145.79.227.38:4978 , piece_id=3JYB42IYDMMSEFRJFAIOWPCV5JJAQOILCEGLHMSBWSXT6GPFNWSA, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.203940152 (260h49m29.075023197s) [noise=true , address=145.79.227.3:4150 , piece_id=WXZYNAKPIHDYYMWUZKBI4G7PP4EV2Y6XDQSADOJZASEBMXL7FXQA, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.347406548 (260h49m29.218489583s) [noise=true , address=77.90.31.17:4773 , piece_id=SN2X36NEBJIJEDXSLJAROMD3WGOT7DHSC6TGCI4BM7EVRUM777OQ, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.361791931 (260h49m29.232874966s) [noise=true , address=45.91.116.9:4264 , piece_id=AYTPESZCOSQFBGMYUSHEQC7YEWWZFZFAP6D4F2Y4ITRHGGSIOHAQ, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.447032213 (260h49m29.318115248s) [noise=true , address=45.91.116.5:4297 , piece_id=GRQJKJS4EIM7A5ATMYUSOIFYZTBBKYO72IITZLO72JDM3BKINXQQ, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.932181896 (260h49m29.803264931s) [noise=true , address=77.90.31.13:4629 , piece_id=QV4CXCCUBF6COTIJRYMXMDN2U2MXBHKKFABXAM5C6AZV3XLRV7JQ, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.932199990 (260h49m29.803283025s) [seg_pos={0 0}, upload=228688]: Segment reached optimal threshold of 13 pieces. Dec 16 12:18:29.932354895 (260h49m29.803437931s) [noise=true , address=45.91.116.6:4975 , piece_id=DNIYXILY5EQJVYJSQUPYSCFJPM7UWDRWEW37ES2FGJTIVIFHQALA, seg_pos={0 0}, upload=228688]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 16 12:18:29.932360746 (260h49m29.803443782s) [noise=true , address=45.91.116.6:4975 , piece_id=DNIYXILY5EQJVYJSQUPYSCFJPM7UWDRWEW37ES2FGJTIVIFHQALA, seg_pos={0 0}, upload=228688]: Long tail cancellation due to optimal threshold Dec 16 12:18:29.932609250 (260h49m29.803692295s) [noise=true , address=77.90.31.14:4664 , piece_id=OLEOUQUB6UYPXDQDFR45ZUXDCISMBCZFEZY7CEYLGW4ISZFKLRAQ, seg_pos={0 0}, upload=228688]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9): protocol: read tcp 188.166.225.170:44270->77.90.31.14:4664: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 16 12:18:29.932616132 (260h49m29.803699167s) [noise=true , address=77.90.31.14:4664 , piece_id=OLEOUQUB6UYPXDQDFR45ZUXDCISMBCZFEZY7CEYLGW4ISZFKLRAQ, seg_pos={0 0}, upload=228688]: Long tail cancellation due to optimal threshold Dec 16 12:18:29.932632554 (260h49m29.803715589s) [noise=true , address=77.90.31.19:4842 , piece_id=ZRYPL7FZVCKKNPRLJKV4FIONAKEK7HUDUFRIGF42TVTOHH7KLGEQ, seg_pos={0 0}, upload=228688]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Dec 16 12:18:29.932650127 (260h49m29.803733172s) [noise=true , address=77.90.31.19:4842 , piece_id=ZRYPL7FZVCKKNPRLJKV4FIONAKEK7HUDUFRIGF42TVTOHH7KLGEQ, seg_pos={0 0}, upload=228688]: Long tail cancellation due to optimal threshold Dec 16 12:18:29.932678501 (260h49m29.803761546s) [seg_pos={0 0}, upload=228688]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 12:18:29.932783451 (260h49m29.803866496s) [upload=228688]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 12:18:29.932789533 (260h49m29.803872578s) [upload=228688]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 12:18:30.222615735 (260h49m30.09369879s) [upload=228688]: Done uploading segments... Dec 16 12:18:30.222639069 (260h49m30.093722344s) [upload=228688]: Upload finished. err: <nil> Duration: 5.632264685s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................................XX.......................] duration: 5.978327383s trace: https://jaeger.dev.storj.io/trace/61b21d2c05e36365 FAIL (2025-12-16 12:35:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 16 12:18:26.867577534 (260h49m26.73866058s) [upload=228688]: Starting upload Dec 16 12:18:26.867632178 (260h49m26.738715213s) [upload=228688]: Done starting upload Dec 16 12:18:26.867824475 (260h49m26.73890751s) [upload=228688]: Uploading segments... Dec 16 12:18:26.868036498 (260h49m26.739119533s) [upload=228688]: Got next segment. Inline: false Dec 16 12:18:26.868072116 (260h49m26.739155151s) [upload=228688]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 12:18:26.868076705 (260h49m26.73915975s) [upload=228688]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 12:18:27.112751871 (260h49m26.983834906s) [seg_pos={0 0}, upload=228688]: Begin upload segment... Dec 16 12:18:27.112835951 (260h49m26.983918996s) [seg_pos={0 0}, upload=228688]: Done begin upload segment. Dec 16 12:18:27.112906205 (260h49m26.98398925s) [noise=true , address=145.79.227.38:4978 , piece_id=3JYB42IYDMMSEFRJFAIOWPCV5JJAQOILCEGLHMSBWSXT6GPFNWSA, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.113024209 (260h49m26.984107255s) [noise=true , address=45.91.116.6:4975 , piece_id=DNIYXILY5EQJVYJSQUPYSCFJPM7UWDRWEW37ES2FGJTIVIFHQALA, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.113642677 (260h49m26.984725722s) [noise=true , address=145.79.227.41:4086 , piece_id=D2YPUZK4HQENCLATWCFHFX6FOKRDINRTPRERRUVXX63HMZNP5TVA, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.114458290 (260h49m26.985541335s) [noise=true , address=145.79.227.3:4150 , piece_id=WXZYNAKPIHDYYMWUZKBI4G7PP4EV2Y6XDQSADOJZASEBMXL7FXQA, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.114469862 (260h49m26.985552907s) [noise=true , address=45.91.116.5:4297 , piece_id=GRQJKJS4EIM7A5ATMYUSOIFYZTBBKYO72IITZLO72JDM3BKINXQQ, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.114980304 (260h49m26.986063339s) [noise=true , address=145.79.227.4:4185 , piece_id=575AA6RIXXODXK2OVQGSSMD4PWPFOXW6J6MSKIQBK33QHIONZ7XA, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.115007396 (260h49m26.986090431s) [noise=true , address=145.79.227.22:4510 , piece_id=JIZVI6JLUC24RSB2FD4AG6DFWY2W243ZDFVEBXAX6WQ77CUBYULA, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.115371549 (260h49m26.986454594s) [noise=true , address=145.79.227.2:4114 , piece_id=QIU7OSO67RMZPWXBCIVCBFVXNBFZ4MMXEUON2EW3U273W6O4XN4Q, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.115802831 (260h49m26.986885866s) [noise=true , address=77.90.31.13:4629 , piece_id=QV4CXCCUBF6COTIJRYMXMDN2U2MXBHKKFABXAM5C6AZV3XLRV7JQ, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.116457888 (260h49m26.987540923s) [noise=true , address=145.79.227.8:4331 , piece_id=CB7Q2YLIVRLUH5QIFSA3WKICUYRJQVQ5J6SRILARC72SWTMUI7LA, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.116866826 (260h49m26.987949861s) [noise=true , address=145.79.227.28:4726 , piece_id=SUMLGDF2DGXG2MXVBYRMGBW6O2V676MQFGATY6RMPAKKHVVELYHA, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.117376858 (260h49m26.988459893s) [noise=true , address=45.91.116.9:4264 , piece_id=AYTPESZCOSQFBGMYUSHEQC7YEWWZFZFAP6D4F2Y4ITRHGGSIOHAQ, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.117883373 (260h49m26.988966408s) [noise=true , address=145.79.227.20:4439 , piece_id=UMDFN6JIAD7FPWBZJKXGWCOOTHR7OHHZNUBXVG5H3Q4X2FG6FAWQ, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.118506488 (260h49m26.989589533s) [noise=true , address=77.90.31.14:4664 , piece_id=OLEOUQUB6UYPXDQDFR45ZUXDCISMBCZFEZY7CEYLGW4ISZFKLRAQ, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.119093947 (260h49m26.990176992s) [noise=true , address=77.90.31.19:4842 , piece_id=ZRYPL7FZVCKKNPRLJKV4FIONAKEK7HUDUFRIGF42TVTOHH7KLGEQ, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.120217387 (260h49m26.991300422s) [noise=true , address=77.90.31.17:4773 , piece_id=SN2X36NEBJIJEDXSLJAROMD3WGOT7DHSC6TGCI4BM7EVRUM777OQ, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:28.723929985 (260h49m28.59501302s) [upload=228688]: Next returned nil segment Dec 16 12:18:28.723956255 (260h49m28.59503929s) [upload=228688]: Waiting for error group managing segments... Dec 16 12:18:29.110968427 (260h49m28.982051472s) [noise=true , address=145.79.227.8:4331 , piece_id=CB7Q2YLIVRLUH5QIFSA3WKICUYRJQVQ5J6SRILARC72SWTMUI7LA, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.112846964 (260h49m28.983929999s) [noise=true , address=145.79.227.41:4086 , piece_id=D2YPUZK4HQENCLATWCFHFX6FOKRDINRTPRERRUVXX63HMZNP5TVA, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.115361984 (260h49m28.986445029s) [noise=true , address=145.79.227.22:4510 , piece_id=JIZVI6JLUC24RSB2FD4AG6DFWY2W243ZDFVEBXAX6WQ77CUBYULA, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.122097569 (260h49m28.993180614s) [noise=true , address=145.79.227.2:4114 , piece_id=QIU7OSO67RMZPWXBCIVCBFVXNBFZ4MMXEUON2EW3U273W6O4XN4Q, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.127395447 (260h49m28.998478482s) [noise=true , address=145.79.227.28:4726 , piece_id=SUMLGDF2DGXG2MXVBYRMGBW6O2V676MQFGATY6RMPAKKHVVELYHA, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.131068261 (260h49m29.002151306s) [noise=true , address=145.79.227.20:4439 , piece_id=UMDFN6JIAD7FPWBZJKXGWCOOTHR7OHHZNUBXVG5H3Q4X2FG6FAWQ, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.179622336 (260h49m29.050705371s) [noise=true , address=145.79.227.4:4185 , piece_id=575AA6RIXXODXK2OVQGSSMD4PWPFOXW6J6MSKIQBK33QHIONZ7XA, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.198721144 (260h49m29.069804179s) [noise=true , address=145.79.227.38:4978 , piece_id=3JYB42IYDMMSEFRJFAIOWPCV5JJAQOILCEGLHMSBWSXT6GPFNWSA, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.203940152 (260h49m29.075023197s) [noise=true , address=145.79.227.3:4150 , piece_id=WXZYNAKPIHDYYMWUZKBI4G7PP4EV2Y6XDQSADOJZASEBMXL7FXQA, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.347406548 (260h49m29.218489583s) [noise=true , address=77.90.31.17:4773 , piece_id=SN2X36NEBJIJEDXSLJAROMD3WGOT7DHSC6TGCI4BM7EVRUM777OQ, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.361791931 (260h49m29.232874966s) [noise=true , address=45.91.116.9:4264 , piece_id=AYTPESZCOSQFBGMYUSHEQC7YEWWZFZFAP6D4F2Y4ITRHGGSIOHAQ, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.447032213 (260h49m29.318115248s) [noise=true , address=45.91.116.5:4297 , piece_id=GRQJKJS4EIM7A5ATMYUSOIFYZTBBKYO72IITZLO72JDM3BKINXQQ, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.932181896 (260h49m29.803264931s) [noise=true , address=77.90.31.13:4629 , piece_id=QV4CXCCUBF6COTIJRYMXMDN2U2MXBHKKFABXAM5C6AZV3XLRV7JQ, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.932199990 (260h49m29.803283025s) [seg_pos={0 0}, upload=228688]: Segment reached optimal threshold of 13 pieces. Dec 16 12:18:29.932354895 (260h49m29.803437931s) [noise=true , address=45.91.116.6:4975 , piece_id=DNIYXILY5EQJVYJSQUPYSCFJPM7UWDRWEW37ES2FGJTIVIFHQALA, seg_pos={0 0}, upload=228688]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 16 12:18:29.932360746 (260h49m29.803443782s) [noise=true , address=45.91.116.6:4975 , piece_id=DNIYXILY5EQJVYJSQUPYSCFJPM7UWDRWEW37ES2FGJTIVIFHQALA, seg_pos={0 0}, upload=228688]: Long tail cancellation due to optimal threshold Dec 16 12:18:29.932609250 (260h49m29.803692295s) [noise=true , address=77.90.31.14:4664 , piece_id=OLEOUQUB6UYPXDQDFR45ZUXDCISMBCZFEZY7CEYLGW4ISZFKLRAQ, seg_pos={0 0}, upload=228688]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9): protocol: read tcp 188.166.225.170:44270->77.90.31.14:4664: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 16 12:18:29.932616132 (260h49m29.803699167s) [noise=true , address=77.90.31.14:4664 , piece_id=OLEOUQUB6UYPXDQDFR45ZUXDCISMBCZFEZY7CEYLGW4ISZFKLRAQ, seg_pos={0 0}, upload=228688]: Long tail cancellation due to optimal threshold Dec 16 12:18:29.932632554 (260h49m29.803715589s) [noise=true , address=77.90.31.19:4842 , piece_id=ZRYPL7FZVCKKNPRLJKV4FIONAKEK7HUDUFRIGF42TVTOHH7KLGEQ, seg_pos={0 0}, upload=228688]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Dec 16 12:18:29.932650127 (260h49m29.803733172s) [noise=true , address=77.90.31.19:4842 , piece_id=ZRYPL7FZVCKKNPRLJKV4FIONAKEK7HUDUFRIGF42TVTOHH7KLGEQ, seg_pos={0 0}, upload=228688]: Long tail cancellation due to optimal threshold Dec 16 12:18:29.932678501 (260h49m29.803761546s) [seg_pos={0 0}, upload=228688]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 12:18:29.932783451 (260h49m29.803866496s) [upload=228688]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 12:18:29.932789533 (260h49m29.803872578s) [upload=228688]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 12:18:30.222615735 (260h49m30.09369879s) [upload=228688]: Done uploading segments... Dec 16 12:18:30.222639069 (260h49m30.093722344s) [upload=228688]: Upload finished. err: <nil> Duration: 5.97835682s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 14 out of 60 is failed (threshold=6) [........X....X.XX.......X....X...XXX......X.X...X........X.X] duration: 5.001880268s trace: https://jaeger.dev.storj.io/trace/42853ca58c898904 FAIL (2025-12-16 12:35:29 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001905556s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................................XX......................] duration: 5.969504627s trace: https://jaeger.dev.storj.io/trace/61b21d2c05e36365 FAIL (2025-12-16 12:35:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 16 12:18:26.867577534 (260h49m26.73866058s) [upload=228688]: Starting upload Dec 16 12:18:26.867632178 (260h49m26.738715213s) [upload=228688]: Done starting upload Dec 16 12:18:26.867824475 (260h49m26.73890751s) [upload=228688]: Uploading segments... Dec 16 12:18:26.868036498 (260h49m26.739119533s) [upload=228688]: Got next segment. Inline: false Dec 16 12:18:26.868072116 (260h49m26.739155151s) [upload=228688]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 12:18:26.868076705 (260h49m26.73915975s) [upload=228688]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 12:18:27.112751871 (260h49m26.983834906s) [seg_pos={0 0}, upload=228688]: Begin upload segment... Dec 16 12:18:27.112835951 (260h49m26.983918996s) [seg_pos={0 0}, upload=228688]: Done begin upload segment. Dec 16 12:18:27.112906205 (260h49m26.98398925s) [noise=true , address=145.79.227.38:4978 , piece_id=3JYB42IYDMMSEFRJFAIOWPCV5JJAQOILCEGLHMSBWSXT6GPFNWSA, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.113024209 (260h49m26.984107255s) [noise=true , address=45.91.116.6:4975 , piece_id=DNIYXILY5EQJVYJSQUPYSCFJPM7UWDRWEW37ES2FGJTIVIFHQALA, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.113642677 (260h49m26.984725722s) [noise=true , address=145.79.227.41:4086 , piece_id=D2YPUZK4HQENCLATWCFHFX6FOKRDINRTPRERRUVXX63HMZNP5TVA, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.114458290 (260h49m26.985541335s) [noise=true , address=145.79.227.3:4150 , piece_id=WXZYNAKPIHDYYMWUZKBI4G7PP4EV2Y6XDQSADOJZASEBMXL7FXQA, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.114469862 (260h49m26.985552907s) [noise=true , address=45.91.116.5:4297 , piece_id=GRQJKJS4EIM7A5ATMYUSOIFYZTBBKYO72IITZLO72JDM3BKINXQQ, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.114980304 (260h49m26.986063339s) [noise=true , address=145.79.227.4:4185 , piece_id=575AA6RIXXODXK2OVQGSSMD4PWPFOXW6J6MSKIQBK33QHIONZ7XA, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.115007396 (260h49m26.986090431s) [noise=true , address=145.79.227.22:4510 , piece_id=JIZVI6JLUC24RSB2FD4AG6DFWY2W243ZDFVEBXAX6WQ77CUBYULA, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.115371549 (260h49m26.986454594s) [noise=true , address=145.79.227.2:4114 , piece_id=QIU7OSO67RMZPWXBCIVCBFVXNBFZ4MMXEUON2EW3U273W6O4XN4Q, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.115802831 (260h49m26.986885866s) [noise=true , address=77.90.31.13:4629 , piece_id=QV4CXCCUBF6COTIJRYMXMDN2U2MXBHKKFABXAM5C6AZV3XLRV7JQ, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.116457888 (260h49m26.987540923s) [noise=true , address=145.79.227.8:4331 , piece_id=CB7Q2YLIVRLUH5QIFSA3WKICUYRJQVQ5J6SRILARC72SWTMUI7LA, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.116866826 (260h49m26.987949861s) [noise=true , address=145.79.227.28:4726 , piece_id=SUMLGDF2DGXG2MXVBYRMGBW6O2V676MQFGATY6RMPAKKHVVELYHA, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.117376858 (260h49m26.988459893s) [noise=true , address=45.91.116.9:4264 , piece_id=AYTPESZCOSQFBGMYUSHEQC7YEWWZFZFAP6D4F2Y4ITRHGGSIOHAQ, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.117883373 (260h49m26.988966408s) [noise=true , address=145.79.227.20:4439 , piece_id=UMDFN6JIAD7FPWBZJKXGWCOOTHR7OHHZNUBXVG5H3Q4X2FG6FAWQ, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.118506488 (260h49m26.989589533s) [noise=true , address=77.90.31.14:4664 , piece_id=OLEOUQUB6UYPXDQDFR45ZUXDCISMBCZFEZY7CEYLGW4ISZFKLRAQ, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.119093947 (260h49m26.990176992s) [noise=true , address=77.90.31.19:4842 , piece_id=ZRYPL7FZVCKKNPRLJKV4FIONAKEK7HUDUFRIGF42TVTOHH7KLGEQ, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.120217387 (260h49m26.991300422s) [noise=true , address=77.90.31.17:4773 , piece_id=SN2X36NEBJIJEDXSLJAROMD3WGOT7DHSC6TGCI4BM7EVRUM777OQ, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:28.723929985 (260h49m28.59501302s) [upload=228688]: Next returned nil segment Dec 16 12:18:28.723956255 (260h49m28.59503929s) [upload=228688]: Waiting for error group managing segments... Dec 16 12:18:29.110968427 (260h49m28.982051472s) [noise=true , address=145.79.227.8:4331 , piece_id=CB7Q2YLIVRLUH5QIFSA3WKICUYRJQVQ5J6SRILARC72SWTMUI7LA, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.112846964 (260h49m28.983929999s) [noise=true , address=145.79.227.41:4086 , piece_id=D2YPUZK4HQENCLATWCFHFX6FOKRDINRTPRERRUVXX63HMZNP5TVA, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.115361984 (260h49m28.986445029s) [noise=true , address=145.79.227.22:4510 , piece_id=JIZVI6JLUC24RSB2FD4AG6DFWY2W243ZDFVEBXAX6WQ77CUBYULA, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.122097569 (260h49m28.993180614s) [noise=true , address=145.79.227.2:4114 , piece_id=QIU7OSO67RMZPWXBCIVCBFVXNBFZ4MMXEUON2EW3U273W6O4XN4Q, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.127395447 (260h49m28.998478482s) [noise=true , address=145.79.227.28:4726 , piece_id=SUMLGDF2DGXG2MXVBYRMGBW6O2V676MQFGATY6RMPAKKHVVELYHA, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.131068261 (260h49m29.002151306s) [noise=true , address=145.79.227.20:4439 , piece_id=UMDFN6JIAD7FPWBZJKXGWCOOTHR7OHHZNUBXVG5H3Q4X2FG6FAWQ, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.179622336 (260h49m29.050705371s) [noise=true , address=145.79.227.4:4185 , piece_id=575AA6RIXXODXK2OVQGSSMD4PWPFOXW6J6MSKIQBK33QHIONZ7XA, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.198721144 (260h49m29.069804179s) [noise=true , address=145.79.227.38:4978 , piece_id=3JYB42IYDMMSEFRJFAIOWPCV5JJAQOILCEGLHMSBWSXT6GPFNWSA, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.203940152 (260h49m29.075023197s) [noise=true , address=145.79.227.3:4150 , piece_id=WXZYNAKPIHDYYMWUZKBI4G7PP4EV2Y6XDQSADOJZASEBMXL7FXQA, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.347406548 (260h49m29.218489583s) [noise=true , address=77.90.31.17:4773 , piece_id=SN2X36NEBJIJEDXSLJAROMD3WGOT7DHSC6TGCI4BM7EVRUM777OQ, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.361791931 (260h49m29.232874966s) [noise=true , address=45.91.116.9:4264 , piece_id=AYTPESZCOSQFBGMYUSHEQC7YEWWZFZFAP6D4F2Y4ITRHGGSIOHAQ, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.447032213 (260h49m29.318115248s) [noise=true , address=45.91.116.5:4297 , piece_id=GRQJKJS4EIM7A5ATMYUSOIFYZTBBKYO72IITZLO72JDM3BKINXQQ, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.932181896 (260h49m29.803264931s) [noise=true , address=77.90.31.13:4629 , piece_id=QV4CXCCUBF6COTIJRYMXMDN2U2MXBHKKFABXAM5C6AZV3XLRV7JQ, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.932199990 (260h49m29.803283025s) [seg_pos={0 0}, upload=228688]: Segment reached optimal threshold of 13 pieces. Dec 16 12:18:29.932354895 (260h49m29.803437931s) [noise=true , address=45.91.116.6:4975 , piece_id=DNIYXILY5EQJVYJSQUPYSCFJPM7UWDRWEW37ES2FGJTIVIFHQALA, seg_pos={0 0}, upload=228688]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 16 12:18:29.932360746 (260h49m29.803443782s) [noise=true , address=45.91.116.6:4975 , piece_id=DNIYXILY5EQJVYJSQUPYSCFJPM7UWDRWEW37ES2FGJTIVIFHQALA, seg_pos={0 0}, upload=228688]: Long tail cancellation due to optimal threshold Dec 16 12:18:29.932609250 (260h49m29.803692295s) [noise=true , address=77.90.31.14:4664 , piece_id=OLEOUQUB6UYPXDQDFR45ZUXDCISMBCZFEZY7CEYLGW4ISZFKLRAQ, seg_pos={0 0}, upload=228688]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9): protocol: read tcp 188.166.225.170:44270->77.90.31.14:4664: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 16 12:18:29.932616132 (260h49m29.803699167s) [noise=true , address=77.90.31.14:4664 , piece_id=OLEOUQUB6UYPXDQDFR45ZUXDCISMBCZFEZY7CEYLGW4ISZFKLRAQ, seg_pos={0 0}, upload=228688]: Long tail cancellation due to optimal threshold Dec 16 12:18:29.932632554 (260h49m29.803715589s) [noise=true , address=77.90.31.19:4842 , piece_id=ZRYPL7FZVCKKNPRLJKV4FIONAKEK7HUDUFRIGF42TVTOHH7KLGEQ, seg_pos={0 0}, upload=228688]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Dec 16 12:18:29.932650127 (260h49m29.803733172s) [noise=true , address=77.90.31.19:4842 , piece_id=ZRYPL7FZVCKKNPRLJKV4FIONAKEK7HUDUFRIGF42TVTOHH7KLGEQ, seg_pos={0 0}, upload=228688]: Long tail cancellation due to optimal threshold Dec 16 12:18:29.932678501 (260h49m29.803761546s) [seg_pos={0 0}, upload=228688]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 12:18:29.932783451 (260h49m29.803866496s) [upload=228688]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 12:18:29.932789533 (260h49m29.803872578s) [upload=228688]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 12:18:30.222615735 (260h49m30.09369879s) [upload=228688]: Done uploading segments... Dec 16 12:18:30.222639069 (260h49m30.093722344s) [upload=228688]: Upload finished. err: <nil> Duration: 5.969532239s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 13 out of 60 is failed (threshold=6) [.........X....X.XX.......X....X...XXX......X.X...X........X.] duration: 3.664776276s trace: https://jaeger.dev.storj.io/trace/41efdd27ad43f746 FAIL (2025-12-16 12:34:55 UTC) orbiter warning: context deadline exceeded Duration: 3.664802907s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 14 out of 60 is failed (threshold=6) [X.........X....X.XX.......X....X...XXX......X.X...X........X] duration: 5.001247313s trace: https://jaeger.dev.storj.io/trace/41efdd27ad43f746 FAIL (2025-12-16 12:34:21 UTC) orbiter warning: context deadline exceeded Duration: 5.001269164s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................................XX.....................] duration: 6.287132573s trace: https://jaeger.dev.storj.io/trace/61b21d2c05e36365 FAIL (2025-12-16 12:34:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 16 12:18:26.867577534 (260h49m26.73866058s) [upload=228688]: Starting upload Dec 16 12:18:26.867632178 (260h49m26.738715213s) [upload=228688]: Done starting upload Dec 16 12:18:26.867824475 (260h49m26.73890751s) [upload=228688]: Uploading segments... Dec 16 12:18:26.868036498 (260h49m26.739119533s) [upload=228688]: Got next segment. Inline: false Dec 16 12:18:26.868072116 (260h49m26.739155151s) [upload=228688]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 12:18:26.868076705 (260h49m26.73915975s) [upload=228688]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 12:18:27.112751871 (260h49m26.983834906s) [seg_pos={0 0}, upload=228688]: Begin upload segment... Dec 16 12:18:27.112835951 (260h49m26.983918996s) [seg_pos={0 0}, upload=228688]: Done begin upload segment. Dec 16 12:18:27.112906205 (260h49m26.98398925s) [noise=true , address=145.79.227.38:4978 , piece_id=3JYB42IYDMMSEFRJFAIOWPCV5JJAQOILCEGLHMSBWSXT6GPFNWSA, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.113024209 (260h49m26.984107255s) [noise=true , address=45.91.116.6:4975 , piece_id=DNIYXILY5EQJVYJSQUPYSCFJPM7UWDRWEW37ES2FGJTIVIFHQALA, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.113642677 (260h49m26.984725722s) [noise=true , address=145.79.227.41:4086 , piece_id=D2YPUZK4HQENCLATWCFHFX6FOKRDINRTPRERRUVXX63HMZNP5TVA, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.114458290 (260h49m26.985541335s) [noise=true , address=145.79.227.3:4150 , piece_id=WXZYNAKPIHDYYMWUZKBI4G7PP4EV2Y6XDQSADOJZASEBMXL7FXQA, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.114469862 (260h49m26.985552907s) [noise=true , address=45.91.116.5:4297 , piece_id=GRQJKJS4EIM7A5ATMYUSOIFYZTBBKYO72IITZLO72JDM3BKINXQQ, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.114980304 (260h49m26.986063339s) [noise=true , address=145.79.227.4:4185 , piece_id=575AA6RIXXODXK2OVQGSSMD4PWPFOXW6J6MSKIQBK33QHIONZ7XA, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.115007396 (260h49m26.986090431s) [noise=true , address=145.79.227.22:4510 , piece_id=JIZVI6JLUC24RSB2FD4AG6DFWY2W243ZDFVEBXAX6WQ77CUBYULA, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.115371549 (260h49m26.986454594s) [noise=true , address=145.79.227.2:4114 , piece_id=QIU7OSO67RMZPWXBCIVCBFVXNBFZ4MMXEUON2EW3U273W6O4XN4Q, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.115802831 (260h49m26.986885866s) [noise=true , address=77.90.31.13:4629 , piece_id=QV4CXCCUBF6COTIJRYMXMDN2U2MXBHKKFABXAM5C6AZV3XLRV7JQ, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.116457888 (260h49m26.987540923s) [noise=true , address=145.79.227.8:4331 , piece_id=CB7Q2YLIVRLUH5QIFSA3WKICUYRJQVQ5J6SRILARC72SWTMUI7LA, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.116866826 (260h49m26.987949861s) [noise=true , address=145.79.227.28:4726 , piece_id=SUMLGDF2DGXG2MXVBYRMGBW6O2V676MQFGATY6RMPAKKHVVELYHA, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.117376858 (260h49m26.988459893s) [noise=true , address=45.91.116.9:4264 , piece_id=AYTPESZCOSQFBGMYUSHEQC7YEWWZFZFAP6D4F2Y4ITRHGGSIOHAQ, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.117883373 (260h49m26.988966408s) [noise=true , address=145.79.227.20:4439 , piece_id=UMDFN6JIAD7FPWBZJKXGWCOOTHR7OHHZNUBXVG5H3Q4X2FG6FAWQ, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.118506488 (260h49m26.989589533s) [noise=true , address=77.90.31.14:4664 , piece_id=OLEOUQUB6UYPXDQDFR45ZUXDCISMBCZFEZY7CEYLGW4ISZFKLRAQ, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.119093947 (260h49m26.990176992s) [noise=true , address=77.90.31.19:4842 , piece_id=ZRYPL7FZVCKKNPRLJKV4FIONAKEK7HUDUFRIGF42TVTOHH7KLGEQ, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.120217387 (260h49m26.991300422s) [noise=true , address=77.90.31.17:4773 , piece_id=SN2X36NEBJIJEDXSLJAROMD3WGOT7DHSC6TGCI4BM7EVRUM777OQ, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:28.723929985 (260h49m28.59501302s) [upload=228688]: Next returned nil segment Dec 16 12:18:28.723956255 (260h49m28.59503929s) [upload=228688]: Waiting for error group managing segments... Dec 16 12:18:29.110968427 (260h49m28.982051472s) [noise=true , address=145.79.227.8:4331 , piece_id=CB7Q2YLIVRLUH5QIFSA3WKICUYRJQVQ5J6SRILARC72SWTMUI7LA, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.112846964 (260h49m28.983929999s) [noise=true , address=145.79.227.41:4086 , piece_id=D2YPUZK4HQENCLATWCFHFX6FOKRDINRTPRERRUVXX63HMZNP5TVA, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.115361984 (260h49m28.986445029s) [noise=true , address=145.79.227.22:4510 , piece_id=JIZVI6JLUC24RSB2FD4AG6DFWY2W243ZDFVEBXAX6WQ77CUBYULA, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.122097569 (260h49m28.993180614s) [noise=true , address=145.79.227.2:4114 , piece_id=QIU7OSO67RMZPWXBCIVCBFVXNBFZ4MMXEUON2EW3U273W6O4XN4Q, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.127395447 (260h49m28.998478482s) [noise=true , address=145.79.227.28:4726 , piece_id=SUMLGDF2DGXG2MXVBYRMGBW6O2V676MQFGATY6RMPAKKHVVELYHA, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.131068261 (260h49m29.002151306s) [noise=true , address=145.79.227.20:4439 , piece_id=UMDFN6JIAD7FPWBZJKXGWCOOTHR7OHHZNUBXVG5H3Q4X2FG6FAWQ, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.179622336 (260h49m29.050705371s) [noise=true , address=145.79.227.4:4185 , piece_id=575AA6RIXXODXK2OVQGSSMD4PWPFOXW6J6MSKIQBK33QHIONZ7XA, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.198721144 (260h49m29.069804179s) [noise=true , address=145.79.227.38:4978 , piece_id=3JYB42IYDMMSEFRJFAIOWPCV5JJAQOILCEGLHMSBWSXT6GPFNWSA, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.203940152 (260h49m29.075023197s) [noise=true , address=145.79.227.3:4150 , piece_id=WXZYNAKPIHDYYMWUZKBI4G7PP4EV2Y6XDQSADOJZASEBMXL7FXQA, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.347406548 (260h49m29.218489583s) [noise=true , address=77.90.31.17:4773 , piece_id=SN2X36NEBJIJEDXSLJAROMD3WGOT7DHSC6TGCI4BM7EVRUM777OQ, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.361791931 (260h49m29.232874966s) [noise=true , address=45.91.116.9:4264 , piece_id=AYTPESZCOSQFBGMYUSHEQC7YEWWZFZFAP6D4F2Y4ITRHGGSIOHAQ, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.447032213 (260h49m29.318115248s) [noise=true , address=45.91.116.5:4297 , piece_id=GRQJKJS4EIM7A5ATMYUSOIFYZTBBKYO72IITZLO72JDM3BKINXQQ, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.932181896 (260h49m29.803264931s) [noise=true , address=77.90.31.13:4629 , piece_id=QV4CXCCUBF6COTIJRYMXMDN2U2MXBHKKFABXAM5C6AZV3XLRV7JQ, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.932199990 (260h49m29.803283025s) [seg_pos={0 0}, upload=228688]: Segment reached optimal threshold of 13 pieces. Dec 16 12:18:29.932354895 (260h49m29.803437931s) [noise=true , address=45.91.116.6:4975 , piece_id=DNIYXILY5EQJVYJSQUPYSCFJPM7UWDRWEW37ES2FGJTIVIFHQALA, seg_pos={0 0}, upload=228688]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 16 12:18:29.932360746 (260h49m29.803443782s) [noise=true , address=45.91.116.6:4975 , piece_id=DNIYXILY5EQJVYJSQUPYSCFJPM7UWDRWEW37ES2FGJTIVIFHQALA, seg_pos={0 0}, upload=228688]: Long tail cancellation due to optimal threshold Dec 16 12:18:29.932609250 (260h49m29.803692295s) [noise=true , address=77.90.31.14:4664 , piece_id=OLEOUQUB6UYPXDQDFR45ZUXDCISMBCZFEZY7CEYLGW4ISZFKLRAQ, seg_pos={0 0}, upload=228688]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9): protocol: read tcp 188.166.225.170:44270->77.90.31.14:4664: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 16 12:18:29.932616132 (260h49m29.803699167s) [noise=true , address=77.90.31.14:4664 , piece_id=OLEOUQUB6UYPXDQDFR45ZUXDCISMBCZFEZY7CEYLGW4ISZFKLRAQ, seg_pos={0 0}, upload=228688]: Long tail cancellation due to optimal threshold Dec 16 12:18:29.932632554 (260h49m29.803715589s) [noise=true , address=77.90.31.19:4842 , piece_id=ZRYPL7FZVCKKNPRLJKV4FIONAKEK7HUDUFRIGF42TVTOHH7KLGEQ, seg_pos={0 0}, upload=228688]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Dec 16 12:18:29.932650127 (260h49m29.803733172s) [noise=true , address=77.90.31.19:4842 , piece_id=ZRYPL7FZVCKKNPRLJKV4FIONAKEK7HUDUFRIGF42TVTOHH7KLGEQ, seg_pos={0 0}, upload=228688]: Long tail cancellation due to optimal threshold Dec 16 12:18:29.932678501 (260h49m29.803761546s) [seg_pos={0 0}, upload=228688]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 12:18:29.932783451 (260h49m29.803866496s) [upload=228688]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 12:18:29.932789533 (260h49m29.803872578s) [upload=228688]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 12:18:30.222615735 (260h49m30.09369879s) [upload=228688]: Done uploading segments... Dec 16 12:18:30.222639069 (260h49m30.093722344s) [upload=228688]: Upload finished. err: <nil> Duration: 6.287190363s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 14 out of 60 is failed (threshold=6) [XX.........X....X.XX.......X....X...XXX......X.X...X........] duration: 4.029346654s trace: https://jaeger.dev.storj.io/trace/531b60cce851d866 FAIL (2025-12-16 12:33:35 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.029368787s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................................XX....................] duration: 6.061759462s trace: https://jaeger.dev.storj.io/trace/61b21d2c05e36365 FAIL (2025-12-16 12:33:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 16 12:18:26.867577534 (260h49m26.73866058s) [upload=228688]: Starting upload Dec 16 12:18:26.867632178 (260h49m26.738715213s) [upload=228688]: Done starting upload Dec 16 12:18:26.867824475 (260h49m26.73890751s) [upload=228688]: Uploading segments... Dec 16 12:18:26.868036498 (260h49m26.739119533s) [upload=228688]: Got next segment. Inline: false Dec 16 12:18:26.868072116 (260h49m26.739155151s) [upload=228688]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 12:18:26.868076705 (260h49m26.73915975s) [upload=228688]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 12:18:27.112751871 (260h49m26.983834906s) [seg_pos={0 0}, upload=228688]: Begin upload segment... Dec 16 12:18:27.112835951 (260h49m26.983918996s) [seg_pos={0 0}, upload=228688]: Done begin upload segment. Dec 16 12:18:27.112906205 (260h49m26.98398925s) [noise=true , address=145.79.227.38:4978 , piece_id=3JYB42IYDMMSEFRJFAIOWPCV5JJAQOILCEGLHMSBWSXT6GPFNWSA, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.113024209 (260h49m26.984107255s) [noise=true , address=45.91.116.6:4975 , piece_id=DNIYXILY5EQJVYJSQUPYSCFJPM7UWDRWEW37ES2FGJTIVIFHQALA, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.113642677 (260h49m26.984725722s) [noise=true , address=145.79.227.41:4086 , piece_id=D2YPUZK4HQENCLATWCFHFX6FOKRDINRTPRERRUVXX63HMZNP5TVA, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.114458290 (260h49m26.985541335s) [noise=true , address=145.79.227.3:4150 , piece_id=WXZYNAKPIHDYYMWUZKBI4G7PP4EV2Y6XDQSADOJZASEBMXL7FXQA, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.114469862 (260h49m26.985552907s) [noise=true , address=45.91.116.5:4297 , piece_id=GRQJKJS4EIM7A5ATMYUSOIFYZTBBKYO72IITZLO72JDM3BKINXQQ, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.114980304 (260h49m26.986063339s) [noise=true , address=145.79.227.4:4185 , piece_id=575AA6RIXXODXK2OVQGSSMD4PWPFOXW6J6MSKIQBK33QHIONZ7XA, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.115007396 (260h49m26.986090431s) [noise=true , address=145.79.227.22:4510 , piece_id=JIZVI6JLUC24RSB2FD4AG6DFWY2W243ZDFVEBXAX6WQ77CUBYULA, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.115371549 (260h49m26.986454594s) [noise=true , address=145.79.227.2:4114 , piece_id=QIU7OSO67RMZPWXBCIVCBFVXNBFZ4MMXEUON2EW3U273W6O4XN4Q, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.115802831 (260h49m26.986885866s) [noise=true , address=77.90.31.13:4629 , piece_id=QV4CXCCUBF6COTIJRYMXMDN2U2MXBHKKFABXAM5C6AZV3XLRV7JQ, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.116457888 (260h49m26.987540923s) [noise=true , address=145.79.227.8:4331 , piece_id=CB7Q2YLIVRLUH5QIFSA3WKICUYRJQVQ5J6SRILARC72SWTMUI7LA, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.116866826 (260h49m26.987949861s) [noise=true , address=145.79.227.28:4726 , piece_id=SUMLGDF2DGXG2MXVBYRMGBW6O2V676MQFGATY6RMPAKKHVVELYHA, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.117376858 (260h49m26.988459893s) [noise=true , address=45.91.116.9:4264 , piece_id=AYTPESZCOSQFBGMYUSHEQC7YEWWZFZFAP6D4F2Y4ITRHGGSIOHAQ, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.117883373 (260h49m26.988966408s) [noise=true , address=145.79.227.20:4439 , piece_id=UMDFN6JIAD7FPWBZJKXGWCOOTHR7OHHZNUBXVG5H3Q4X2FG6FAWQ, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.118506488 (260h49m26.989589533s) [noise=true , address=77.90.31.14:4664 , piece_id=OLEOUQUB6UYPXDQDFR45ZUXDCISMBCZFEZY7CEYLGW4ISZFKLRAQ, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.119093947 (260h49m26.990176992s) [noise=true , address=77.90.31.19:4842 , piece_id=ZRYPL7FZVCKKNPRLJKV4FIONAKEK7HUDUFRIGF42TVTOHH7KLGEQ, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.120217387 (260h49m26.991300422s) [noise=true , address=77.90.31.17:4773 , piece_id=SN2X36NEBJIJEDXSLJAROMD3WGOT7DHSC6TGCI4BM7EVRUM777OQ, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:28.723929985 (260h49m28.59501302s) [upload=228688]: Next returned nil segment Dec 16 12:18:28.723956255 (260h49m28.59503929s) [upload=228688]: Waiting for error group managing segments... Dec 16 12:18:29.110968427 (260h49m28.982051472s) [noise=true , address=145.79.227.8:4331 , piece_id=CB7Q2YLIVRLUH5QIFSA3WKICUYRJQVQ5J6SRILARC72SWTMUI7LA, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.112846964 (260h49m28.983929999s) [noise=true , address=145.79.227.41:4086 , piece_id=D2YPUZK4HQENCLATWCFHFX6FOKRDINRTPRERRUVXX63HMZNP5TVA, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.115361984 (260h49m28.986445029s) [noise=true , address=145.79.227.22:4510 , piece_id=JIZVI6JLUC24RSB2FD4AG6DFWY2W243ZDFVEBXAX6WQ77CUBYULA, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.122097569 (260h49m28.993180614s) [noise=true , address=145.79.227.2:4114 , piece_id=QIU7OSO67RMZPWXBCIVCBFVXNBFZ4MMXEUON2EW3U273W6O4XN4Q, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.127395447 (260h49m28.998478482s) [noise=true , address=145.79.227.28:4726 , piece_id=SUMLGDF2DGXG2MXVBYRMGBW6O2V676MQFGATY6RMPAKKHVVELYHA, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.131068261 (260h49m29.002151306s) [noise=true , address=145.79.227.20:4439 , piece_id=UMDFN6JIAD7FPWBZJKXGWCOOTHR7OHHZNUBXVG5H3Q4X2FG6FAWQ, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.179622336 (260h49m29.050705371s) [noise=true , address=145.79.227.4:4185 , piece_id=575AA6RIXXODXK2OVQGSSMD4PWPFOXW6J6MSKIQBK33QHIONZ7XA, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.198721144 (260h49m29.069804179s) [noise=true , address=145.79.227.38:4978 , piece_id=3JYB42IYDMMSEFRJFAIOWPCV5JJAQOILCEGLHMSBWSXT6GPFNWSA, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.203940152 (260h49m29.075023197s) [noise=true , address=145.79.227.3:4150 , piece_id=WXZYNAKPIHDYYMWUZKBI4G7PP4EV2Y6XDQSADOJZASEBMXL7FXQA, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.347406548 (260h49m29.218489583s) [noise=true , address=77.90.31.17:4773 , piece_id=SN2X36NEBJIJEDXSLJAROMD3WGOT7DHSC6TGCI4BM7EVRUM777OQ, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.361791931 (260h49m29.232874966s) [noise=true , address=45.91.116.9:4264 , piece_id=AYTPESZCOSQFBGMYUSHEQC7YEWWZFZFAP6D4F2Y4ITRHGGSIOHAQ, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.447032213 (260h49m29.318115248s) [noise=true , address=45.91.116.5:4297 , piece_id=GRQJKJS4EIM7A5ATMYUSOIFYZTBBKYO72IITZLO72JDM3BKINXQQ, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.932181896 (260h49m29.803264931s) [noise=true , address=77.90.31.13:4629 , piece_id=QV4CXCCUBF6COTIJRYMXMDN2U2MXBHKKFABXAM5C6AZV3XLRV7JQ, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.932199990 (260h49m29.803283025s) [seg_pos={0 0}, upload=228688]: Segment reached optimal threshold of 13 pieces. Dec 16 12:18:29.932354895 (260h49m29.803437931s) [noise=true , address=45.91.116.6:4975 , piece_id=DNIYXILY5EQJVYJSQUPYSCFJPM7UWDRWEW37ES2FGJTIVIFHQALA, seg_pos={0 0}, upload=228688]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 16 12:18:29.932360746 (260h49m29.803443782s) [noise=true , address=45.91.116.6:4975 , piece_id=DNIYXILY5EQJVYJSQUPYSCFJPM7UWDRWEW37ES2FGJTIVIFHQALA, seg_pos={0 0}, upload=228688]: Long tail cancellation due to optimal threshold Dec 16 12:18:29.932609250 (260h49m29.803692295s) [noise=true , address=77.90.31.14:4664 , piece_id=OLEOUQUB6UYPXDQDFR45ZUXDCISMBCZFEZY7CEYLGW4ISZFKLRAQ, seg_pos={0 0}, upload=228688]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9): protocol: read tcp 188.166.225.170:44270->77.90.31.14:4664: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 16 12:18:29.932616132 (260h49m29.803699167s) [noise=true , address=77.90.31.14:4664 , piece_id=OLEOUQUB6UYPXDQDFR45ZUXDCISMBCZFEZY7CEYLGW4ISZFKLRAQ, seg_pos={0 0}, upload=228688]: Long tail cancellation due to optimal threshold Dec 16 12:18:29.932632554 (260h49m29.803715589s) [noise=true , address=77.90.31.19:4842 , piece_id=ZRYPL7FZVCKKNPRLJKV4FIONAKEK7HUDUFRIGF42TVTOHH7KLGEQ, seg_pos={0 0}, upload=228688]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Dec 16 12:18:29.932650127 (260h49m29.803733172s) [noise=true , address=77.90.31.19:4842 , piece_id=ZRYPL7FZVCKKNPRLJKV4FIONAKEK7HUDUFRIGF42TVTOHH7KLGEQ, seg_pos={0 0}, upload=228688]: Long tail cancellation due to optimal threshold Dec 16 12:18:29.932678501 (260h49m29.803761546s) [seg_pos={0 0}, upload=228688]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 12:18:29.932783451 (260h49m29.803866496s) [upload=228688]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 12:18:29.932789533 (260h49m29.803872578s) [upload=228688]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 12:18:30.222615735 (260h49m30.09369879s) [upload=228688]: Done uploading segments... Dec 16 12:18:30.222639069 (260h49m30.093722344s) [upload=228688]: Upload finished. err: <nil> Duration: 6.061793696s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 14 out of 60 is failed (threshold=6) [.XX.........X....X.XX.......X....X...XXX......X.X...X.......] duration: 4.519976734s trace: https://jaeger.dev.storj.io/trace/531b60cce851d866 FAIL (2025-12-16 12:32:29 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.520003244s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................................XX...................] duration: 5.769067369s trace: https://jaeger.dev.storj.io/trace/61b21d2c05e36365 FAIL (2025-12-16 12:32:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 16 12:18:26.867577534 (260h49m26.73866058s) [upload=228688]: Starting upload Dec 16 12:18:26.867632178 (260h49m26.738715213s) [upload=228688]: Done starting upload Dec 16 12:18:26.867824475 (260h49m26.73890751s) [upload=228688]: Uploading segments... Dec 16 12:18:26.868036498 (260h49m26.739119533s) [upload=228688]: Got next segment. Inline: false Dec 16 12:18:26.868072116 (260h49m26.739155151s) [upload=228688]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 12:18:26.868076705 (260h49m26.73915975s) [upload=228688]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 12:18:27.112751871 (260h49m26.983834906s) [seg_pos={0 0}, upload=228688]: Begin upload segment... Dec 16 12:18:27.112835951 (260h49m26.983918996s) [seg_pos={0 0}, upload=228688]: Done begin upload segment. Dec 16 12:18:27.112906205 (260h49m26.98398925s) [noise=true , address=145.79.227.38:4978 , piece_id=3JYB42IYDMMSEFRJFAIOWPCV5JJAQOILCEGLHMSBWSXT6GPFNWSA, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.113024209 (260h49m26.984107255s) [noise=true , address=45.91.116.6:4975 , piece_id=DNIYXILY5EQJVYJSQUPYSCFJPM7UWDRWEW37ES2FGJTIVIFHQALA, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.113642677 (260h49m26.984725722s) [noise=true , address=145.79.227.41:4086 , piece_id=D2YPUZK4HQENCLATWCFHFX6FOKRDINRTPRERRUVXX63HMZNP5TVA, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.114458290 (260h49m26.985541335s) [noise=true , address=145.79.227.3:4150 , piece_id=WXZYNAKPIHDYYMWUZKBI4G7PP4EV2Y6XDQSADOJZASEBMXL7FXQA, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.114469862 (260h49m26.985552907s) [noise=true , address=45.91.116.5:4297 , piece_id=GRQJKJS4EIM7A5ATMYUSOIFYZTBBKYO72IITZLO72JDM3BKINXQQ, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.114980304 (260h49m26.986063339s) [noise=true , address=145.79.227.4:4185 , piece_id=575AA6RIXXODXK2OVQGSSMD4PWPFOXW6J6MSKIQBK33QHIONZ7XA, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.115007396 (260h49m26.986090431s) [noise=true , address=145.79.227.22:4510 , piece_id=JIZVI6JLUC24RSB2FD4AG6DFWY2W243ZDFVEBXAX6WQ77CUBYULA, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.115371549 (260h49m26.986454594s) [noise=true , address=145.79.227.2:4114 , piece_id=QIU7OSO67RMZPWXBCIVCBFVXNBFZ4MMXEUON2EW3U273W6O4XN4Q, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.115802831 (260h49m26.986885866s) [noise=true , address=77.90.31.13:4629 , piece_id=QV4CXCCUBF6COTIJRYMXMDN2U2MXBHKKFABXAM5C6AZV3XLRV7JQ, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.116457888 (260h49m26.987540923s) [noise=true , address=145.79.227.8:4331 , piece_id=CB7Q2YLIVRLUH5QIFSA3WKICUYRJQVQ5J6SRILARC72SWTMUI7LA, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.116866826 (260h49m26.987949861s) [noise=true , address=145.79.227.28:4726 , piece_id=SUMLGDF2DGXG2MXVBYRMGBW6O2V676MQFGATY6RMPAKKHVVELYHA, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.117376858 (260h49m26.988459893s) [noise=true , address=45.91.116.9:4264 , piece_id=AYTPESZCOSQFBGMYUSHEQC7YEWWZFZFAP6D4F2Y4ITRHGGSIOHAQ, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.117883373 (260h49m26.988966408s) [noise=true , address=145.79.227.20:4439 , piece_id=UMDFN6JIAD7FPWBZJKXGWCOOTHR7OHHZNUBXVG5H3Q4X2FG6FAWQ, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.118506488 (260h49m26.989589533s) [noise=true , address=77.90.31.14:4664 , piece_id=OLEOUQUB6UYPXDQDFR45ZUXDCISMBCZFEZY7CEYLGW4ISZFKLRAQ, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.119093947 (260h49m26.990176992s) [noise=true , address=77.90.31.19:4842 , piece_id=ZRYPL7FZVCKKNPRLJKV4FIONAKEK7HUDUFRIGF42TVTOHH7KLGEQ, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.120217387 (260h49m26.991300422s) [noise=true , address=77.90.31.17:4773 , piece_id=SN2X36NEBJIJEDXSLJAROMD3WGOT7DHSC6TGCI4BM7EVRUM777OQ, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:28.723929985 (260h49m28.59501302s) [upload=228688]: Next returned nil segment Dec 16 12:18:28.723956255 (260h49m28.59503929s) [upload=228688]: Waiting for error group managing segments... Dec 16 12:18:29.110968427 (260h49m28.982051472s) [noise=true , address=145.79.227.8:4331 , piece_id=CB7Q2YLIVRLUH5QIFSA3WKICUYRJQVQ5J6SRILARC72SWTMUI7LA, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.112846964 (260h49m28.983929999s) [noise=true , address=145.79.227.41:4086 , piece_id=D2YPUZK4HQENCLATWCFHFX6FOKRDINRTPRERRUVXX63HMZNP5TVA, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.115361984 (260h49m28.986445029s) [noise=true , address=145.79.227.22:4510 , piece_id=JIZVI6JLUC24RSB2FD4AG6DFWY2W243ZDFVEBXAX6WQ77CUBYULA, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.122097569 (260h49m28.993180614s) [noise=true , address=145.79.227.2:4114 , piece_id=QIU7OSO67RMZPWXBCIVCBFVXNBFZ4MMXEUON2EW3U273W6O4XN4Q, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.127395447 (260h49m28.998478482s) [noise=true , address=145.79.227.28:4726 , piece_id=SUMLGDF2DGXG2MXVBYRMGBW6O2V676MQFGATY6RMPAKKHVVELYHA, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.131068261 (260h49m29.002151306s) [noise=true , address=145.79.227.20:4439 , piece_id=UMDFN6JIAD7FPWBZJKXGWCOOTHR7OHHZNUBXVG5H3Q4X2FG6FAWQ, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.179622336 (260h49m29.050705371s) [noise=true , address=145.79.227.4:4185 , piece_id=575AA6RIXXODXK2OVQGSSMD4PWPFOXW6J6MSKIQBK33QHIONZ7XA, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.198721144 (260h49m29.069804179s) [noise=true , address=145.79.227.38:4978 , piece_id=3JYB42IYDMMSEFRJFAIOWPCV5JJAQOILCEGLHMSBWSXT6GPFNWSA, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.203940152 (260h49m29.075023197s) [noise=true , address=145.79.227.3:4150 , piece_id=WXZYNAKPIHDYYMWUZKBI4G7PP4EV2Y6XDQSADOJZASEBMXL7FXQA, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.347406548 (260h49m29.218489583s) [noise=true , address=77.90.31.17:4773 , piece_id=SN2X36NEBJIJEDXSLJAROMD3WGOT7DHSC6TGCI4BM7EVRUM777OQ, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.361791931 (260h49m29.232874966s) [noise=true , address=45.91.116.9:4264 , piece_id=AYTPESZCOSQFBGMYUSHEQC7YEWWZFZFAP6D4F2Y4ITRHGGSIOHAQ, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.447032213 (260h49m29.318115248s) [noise=true , address=45.91.116.5:4297 , piece_id=GRQJKJS4EIM7A5ATMYUSOIFYZTBBKYO72IITZLO72JDM3BKINXQQ, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.932181896 (260h49m29.803264931s) [noise=true , address=77.90.31.13:4629 , piece_id=QV4CXCCUBF6COTIJRYMXMDN2U2MXBHKKFABXAM5C6AZV3XLRV7JQ, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.932199990 (260h49m29.803283025s) [seg_pos={0 0}, upload=228688]: Segment reached optimal threshold of 13 pieces. Dec 16 12:18:29.932354895 (260h49m29.803437931s) [noise=true , address=45.91.116.6:4975 , piece_id=DNIYXILY5EQJVYJSQUPYSCFJPM7UWDRWEW37ES2FGJTIVIFHQALA, seg_pos={0 0}, upload=228688]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 16 12:18:29.932360746 (260h49m29.803443782s) [noise=true , address=45.91.116.6:4975 , piece_id=DNIYXILY5EQJVYJSQUPYSCFJPM7UWDRWEW37ES2FGJTIVIFHQALA, seg_pos={0 0}, upload=228688]: Long tail cancellation due to optimal threshold Dec 16 12:18:29.932609250 (260h49m29.803692295s) [noise=true , address=77.90.31.14:4664 , piece_id=OLEOUQUB6UYPXDQDFR45ZUXDCISMBCZFEZY7CEYLGW4ISZFKLRAQ, seg_pos={0 0}, upload=228688]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9): protocol: read tcp 188.166.225.170:44270->77.90.31.14:4664: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 16 12:18:29.932616132 (260h49m29.803699167s) [noise=true , address=77.90.31.14:4664 , piece_id=OLEOUQUB6UYPXDQDFR45ZUXDCISMBCZFEZY7CEYLGW4ISZFKLRAQ, seg_pos={0 0}, upload=228688]: Long tail cancellation due to optimal threshold Dec 16 12:18:29.932632554 (260h49m29.803715589s) [noise=true , address=77.90.31.19:4842 , piece_id=ZRYPL7FZVCKKNPRLJKV4FIONAKEK7HUDUFRIGF42TVTOHH7KLGEQ, seg_pos={0 0}, upload=228688]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Dec 16 12:18:29.932650127 (260h49m29.803733172s) [noise=true , address=77.90.31.19:4842 , piece_id=ZRYPL7FZVCKKNPRLJKV4FIONAKEK7HUDUFRIGF42TVTOHH7KLGEQ, seg_pos={0 0}, upload=228688]: Long tail cancellation due to optimal threshold Dec 16 12:18:29.932678501 (260h49m29.803761546s) [seg_pos={0 0}, upload=228688]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 12:18:29.932783451 (260h49m29.803866496s) [upload=228688]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 12:18:29.932789533 (260h49m29.803872578s) [upload=228688]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 12:18:30.222615735 (260h49m30.09369879s) [upload=228688]: Done uploading segments... Dec 16 12:18:30.222639069 (260h49m30.093722344s) [upload=228688]: Upload finished. err: <nil> Duration: 5.769099141s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 14 out of 60 is failed (threshold=6) [..XX.........X....X.XX.......X....X...XXX......X.X...X......] duration: 5.00168234s trace: https://jaeger.dev.storj.io/trace/531b60cce851d866 FAIL (2025-12-16 12:32:00 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00171886s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 14 out of 60 is failed (threshold=6) [...XX.........X....X.XX.......X....X...XXX......X.X...X.....] duration: 3.763188673s trace: https://jaeger.dev.storj.io/trace/531b60cce851d866 FAIL (2025-12-16 12:31:08 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.763222667s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................................XX..................] duration: 5.877115425s trace: https://jaeger.dev.storj.io/trace/61b21d2c05e36365 FAIL (2025-12-16 12:31:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 16 12:18:26.867577534 (260h49m26.73866058s) [upload=228688]: Starting upload Dec 16 12:18:26.867632178 (260h49m26.738715213s) [upload=228688]: Done starting upload Dec 16 12:18:26.867824475 (260h49m26.73890751s) [upload=228688]: Uploading segments... Dec 16 12:18:26.868036498 (260h49m26.739119533s) [upload=228688]: Got next segment. Inline: false Dec 16 12:18:26.868072116 (260h49m26.739155151s) [upload=228688]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 12:18:26.868076705 (260h49m26.73915975s) [upload=228688]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 12:18:27.112751871 (260h49m26.983834906s) [seg_pos={0 0}, upload=228688]: Begin upload segment... Dec 16 12:18:27.112835951 (260h49m26.983918996s) [seg_pos={0 0}, upload=228688]: Done begin upload segment. Dec 16 12:18:27.112906205 (260h49m26.98398925s) [noise=true , address=145.79.227.38:4978 , piece_id=3JYB42IYDMMSEFRJFAIOWPCV5JJAQOILCEGLHMSBWSXT6GPFNWSA, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.113024209 (260h49m26.984107255s) [noise=true , address=45.91.116.6:4975 , piece_id=DNIYXILY5EQJVYJSQUPYSCFJPM7UWDRWEW37ES2FGJTIVIFHQALA, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.113642677 (260h49m26.984725722s) [noise=true , address=145.79.227.41:4086 , piece_id=D2YPUZK4HQENCLATWCFHFX6FOKRDINRTPRERRUVXX63HMZNP5TVA, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.114458290 (260h49m26.985541335s) [noise=true , address=145.79.227.3:4150 , piece_id=WXZYNAKPIHDYYMWUZKBI4G7PP4EV2Y6XDQSADOJZASEBMXL7FXQA, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.114469862 (260h49m26.985552907s) [noise=true , address=45.91.116.5:4297 , piece_id=GRQJKJS4EIM7A5ATMYUSOIFYZTBBKYO72IITZLO72JDM3BKINXQQ, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.114980304 (260h49m26.986063339s) [noise=true , address=145.79.227.4:4185 , piece_id=575AA6RIXXODXK2OVQGSSMD4PWPFOXW6J6MSKIQBK33QHIONZ7XA, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.115007396 (260h49m26.986090431s) [noise=true , address=145.79.227.22:4510 , piece_id=JIZVI6JLUC24RSB2FD4AG6DFWY2W243ZDFVEBXAX6WQ77CUBYULA, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.115371549 (260h49m26.986454594s) [noise=true , address=145.79.227.2:4114 , piece_id=QIU7OSO67RMZPWXBCIVCBFVXNBFZ4MMXEUON2EW3U273W6O4XN4Q, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.115802831 (260h49m26.986885866s) [noise=true , address=77.90.31.13:4629 , piece_id=QV4CXCCUBF6COTIJRYMXMDN2U2MXBHKKFABXAM5C6AZV3XLRV7JQ, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.116457888 (260h49m26.987540923s) [noise=true , address=145.79.227.8:4331 , piece_id=CB7Q2YLIVRLUH5QIFSA3WKICUYRJQVQ5J6SRILARC72SWTMUI7LA, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.116866826 (260h49m26.987949861s) [noise=true , address=145.79.227.28:4726 , piece_id=SUMLGDF2DGXG2MXVBYRMGBW6O2V676MQFGATY6RMPAKKHVVELYHA, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.117376858 (260h49m26.988459893s) [noise=true , address=45.91.116.9:4264 , piece_id=AYTPESZCOSQFBGMYUSHEQC7YEWWZFZFAP6D4F2Y4ITRHGGSIOHAQ, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.117883373 (260h49m26.988966408s) [noise=true , address=145.79.227.20:4439 , piece_id=UMDFN6JIAD7FPWBZJKXGWCOOTHR7OHHZNUBXVG5H3Q4X2FG6FAWQ, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.118506488 (260h49m26.989589533s) [noise=true , address=77.90.31.14:4664 , piece_id=OLEOUQUB6UYPXDQDFR45ZUXDCISMBCZFEZY7CEYLGW4ISZFKLRAQ, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.119093947 (260h49m26.990176992s) [noise=true , address=77.90.31.19:4842 , piece_id=ZRYPL7FZVCKKNPRLJKV4FIONAKEK7HUDUFRIGF42TVTOHH7KLGEQ, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.120217387 (260h49m26.991300422s) [noise=true , address=77.90.31.17:4773 , piece_id=SN2X36NEBJIJEDXSLJAROMD3WGOT7DHSC6TGCI4BM7EVRUM777OQ, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:28.723929985 (260h49m28.59501302s) [upload=228688]: Next returned nil segment Dec 16 12:18:28.723956255 (260h49m28.59503929s) [upload=228688]: Waiting for error group managing segments... Dec 16 12:18:29.110968427 (260h49m28.982051472s) [noise=true , address=145.79.227.8:4331 , piece_id=CB7Q2YLIVRLUH5QIFSA3WKICUYRJQVQ5J6SRILARC72SWTMUI7LA, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.112846964 (260h49m28.983929999s) [noise=true , address=145.79.227.41:4086 , piece_id=D2YPUZK4HQENCLATWCFHFX6FOKRDINRTPRERRUVXX63HMZNP5TVA, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.115361984 (260h49m28.986445029s) [noise=true , address=145.79.227.22:4510 , piece_id=JIZVI6JLUC24RSB2FD4AG6DFWY2W243ZDFVEBXAX6WQ77CUBYULA, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.122097569 (260h49m28.993180614s) [noise=true , address=145.79.227.2:4114 , piece_id=QIU7OSO67RMZPWXBCIVCBFVXNBFZ4MMXEUON2EW3U273W6O4XN4Q, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.127395447 (260h49m28.998478482s) [noise=true , address=145.79.227.28:4726 , piece_id=SUMLGDF2DGXG2MXVBYRMGBW6O2V676MQFGATY6RMPAKKHVVELYHA, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.131068261 (260h49m29.002151306s) [noise=true , address=145.79.227.20:4439 , piece_id=UMDFN6JIAD7FPWBZJKXGWCOOTHR7OHHZNUBXVG5H3Q4X2FG6FAWQ, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.179622336 (260h49m29.050705371s) [noise=true , address=145.79.227.4:4185 , piece_id=575AA6RIXXODXK2OVQGSSMD4PWPFOXW6J6MSKIQBK33QHIONZ7XA, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.198721144 (260h49m29.069804179s) [noise=true , address=145.79.227.38:4978 , piece_id=3JYB42IYDMMSEFRJFAIOWPCV5JJAQOILCEGLHMSBWSXT6GPFNWSA, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.203940152 (260h49m29.075023197s) [noise=true , address=145.79.227.3:4150 , piece_id=WXZYNAKPIHDYYMWUZKBI4G7PP4EV2Y6XDQSADOJZASEBMXL7FXQA, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.347406548 (260h49m29.218489583s) [noise=true , address=77.90.31.17:4773 , piece_id=SN2X36NEBJIJEDXSLJAROMD3WGOT7DHSC6TGCI4BM7EVRUM777OQ, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.361791931 (260h49m29.232874966s) [noise=true , address=45.91.116.9:4264 , piece_id=AYTPESZCOSQFBGMYUSHEQC7YEWWZFZFAP6D4F2Y4ITRHGGSIOHAQ, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.447032213 (260h49m29.318115248s) [noise=true , address=45.91.116.5:4297 , piece_id=GRQJKJS4EIM7A5ATMYUSOIFYZTBBKYO72IITZLO72JDM3BKINXQQ, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.932181896 (260h49m29.803264931s) [noise=true , address=77.90.31.13:4629 , piece_id=QV4CXCCUBF6COTIJRYMXMDN2U2MXBHKKFABXAM5C6AZV3XLRV7JQ, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.932199990 (260h49m29.803283025s) [seg_pos={0 0}, upload=228688]: Segment reached optimal threshold of 13 pieces. Dec 16 12:18:29.932354895 (260h49m29.803437931s) [noise=true , address=45.91.116.6:4975 , piece_id=DNIYXILY5EQJVYJSQUPYSCFJPM7UWDRWEW37ES2FGJTIVIFHQALA, seg_pos={0 0}, upload=228688]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 16 12:18:29.932360746 (260h49m29.803443782s) [noise=true , address=45.91.116.6:4975 , piece_id=DNIYXILY5EQJVYJSQUPYSCFJPM7UWDRWEW37ES2FGJTIVIFHQALA, seg_pos={0 0}, upload=228688]: Long tail cancellation due to optimal threshold Dec 16 12:18:29.932609250 (260h49m29.803692295s) [noise=true , address=77.90.31.14:4664 , piece_id=OLEOUQUB6UYPXDQDFR45ZUXDCISMBCZFEZY7CEYLGW4ISZFKLRAQ, seg_pos={0 0}, upload=228688]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9): protocol: read tcp 188.166.225.170:44270->77.90.31.14:4664: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 16 12:18:29.932616132 (260h49m29.803699167s) [noise=true , address=77.90.31.14:4664 , piece_id=OLEOUQUB6UYPXDQDFR45ZUXDCISMBCZFEZY7CEYLGW4ISZFKLRAQ, seg_pos={0 0}, upload=228688]: Long tail cancellation due to optimal threshold Dec 16 12:18:29.932632554 (260h49m29.803715589s) [noise=true , address=77.90.31.19:4842 , piece_id=ZRYPL7FZVCKKNPRLJKV4FIONAKEK7HUDUFRIGF42TVTOHH7KLGEQ, seg_pos={0 0}, upload=228688]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Dec 16 12:18:29.932650127 (260h49m29.803733172s) [noise=true , address=77.90.31.19:4842 , piece_id=ZRYPL7FZVCKKNPRLJKV4FIONAKEK7HUDUFRIGF42TVTOHH7KLGEQ, seg_pos={0 0}, upload=228688]: Long tail cancellation due to optimal threshold Dec 16 12:18:29.932678501 (260h49m29.803761546s) [seg_pos={0 0}, upload=228688]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 12:18:29.932783451 (260h49m29.803866496s) [upload=228688]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 12:18:29.932789533 (260h49m29.803872578s) [upload=228688]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 12:18:30.222615735 (260h49m30.09369879s) [upload=228688]: Done uploading segments... Dec 16 12:18:30.222639069 (260h49m30.093722344s) [upload=228688]: Upload finished. err: <nil> Duration: 5.877148048s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 14 out of 60 is failed (threshold=6) [....XX.........X....X.XX.......X....X...XXX......X.X...X....] duration: 4.094808449s trace: https://jaeger.dev.storj.io/trace/531b60cce851d866 FAIL (2025-12-16 12:30:46 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.094843014s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................................XX.................] duration: 6.136154685s trace: https://jaeger.dev.storj.io/trace/61b21d2c05e36365 FAIL (2025-12-16 12:30:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 16 12:18:26.867577534 (260h49m26.73866058s) [upload=228688]: Starting upload Dec 16 12:18:26.867632178 (260h49m26.738715213s) [upload=228688]: Done starting upload Dec 16 12:18:26.867824475 (260h49m26.73890751s) [upload=228688]: Uploading segments... Dec 16 12:18:26.868036498 (260h49m26.739119533s) [upload=228688]: Got next segment. Inline: false Dec 16 12:18:26.868072116 (260h49m26.739155151s) [upload=228688]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 12:18:26.868076705 (260h49m26.73915975s) [upload=228688]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 12:18:27.112751871 (260h49m26.983834906s) [seg_pos={0 0}, upload=228688]: Begin upload segment... Dec 16 12:18:27.112835951 (260h49m26.983918996s) [seg_pos={0 0}, upload=228688]: Done begin upload segment. Dec 16 12:18:27.112906205 (260h49m26.98398925s) [noise=true , address=145.79.227.38:4978 , piece_id=3JYB42IYDMMSEFRJFAIOWPCV5JJAQOILCEGLHMSBWSXT6GPFNWSA, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.113024209 (260h49m26.984107255s) [noise=true , address=45.91.116.6:4975 , piece_id=DNIYXILY5EQJVYJSQUPYSCFJPM7UWDRWEW37ES2FGJTIVIFHQALA, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.113642677 (260h49m26.984725722s) [noise=true , address=145.79.227.41:4086 , piece_id=D2YPUZK4HQENCLATWCFHFX6FOKRDINRTPRERRUVXX63HMZNP5TVA, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.114458290 (260h49m26.985541335s) [noise=true , address=145.79.227.3:4150 , piece_id=WXZYNAKPIHDYYMWUZKBI4G7PP4EV2Y6XDQSADOJZASEBMXL7FXQA, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.114469862 (260h49m26.985552907s) [noise=true , address=45.91.116.5:4297 , piece_id=GRQJKJS4EIM7A5ATMYUSOIFYZTBBKYO72IITZLO72JDM3BKINXQQ, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.114980304 (260h49m26.986063339s) [noise=true , address=145.79.227.4:4185 , piece_id=575AA6RIXXODXK2OVQGSSMD4PWPFOXW6J6MSKIQBK33QHIONZ7XA, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.115007396 (260h49m26.986090431s) [noise=true , address=145.79.227.22:4510 , piece_id=JIZVI6JLUC24RSB2FD4AG6DFWY2W243ZDFVEBXAX6WQ77CUBYULA, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.115371549 (260h49m26.986454594s) [noise=true , address=145.79.227.2:4114 , piece_id=QIU7OSO67RMZPWXBCIVCBFVXNBFZ4MMXEUON2EW3U273W6O4XN4Q, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.115802831 (260h49m26.986885866s) [noise=true , address=77.90.31.13:4629 , piece_id=QV4CXCCUBF6COTIJRYMXMDN2U2MXBHKKFABXAM5C6AZV3XLRV7JQ, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.116457888 (260h49m26.987540923s) [noise=true , address=145.79.227.8:4331 , piece_id=CB7Q2YLIVRLUH5QIFSA3WKICUYRJQVQ5J6SRILARC72SWTMUI7LA, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.116866826 (260h49m26.987949861s) [noise=true , address=145.79.227.28:4726 , piece_id=SUMLGDF2DGXG2MXVBYRMGBW6O2V676MQFGATY6RMPAKKHVVELYHA, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.117376858 (260h49m26.988459893s) [noise=true , address=45.91.116.9:4264 , piece_id=AYTPESZCOSQFBGMYUSHEQC7YEWWZFZFAP6D4F2Y4ITRHGGSIOHAQ, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.117883373 (260h49m26.988966408s) [noise=true , address=145.79.227.20:4439 , piece_id=UMDFN6JIAD7FPWBZJKXGWCOOTHR7OHHZNUBXVG5H3Q4X2FG6FAWQ, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.118506488 (260h49m26.989589533s) [noise=true , address=77.90.31.14:4664 , piece_id=OLEOUQUB6UYPXDQDFR45ZUXDCISMBCZFEZY7CEYLGW4ISZFKLRAQ, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.119093947 (260h49m26.990176992s) [noise=true , address=77.90.31.19:4842 , piece_id=ZRYPL7FZVCKKNPRLJKV4FIONAKEK7HUDUFRIGF42TVTOHH7KLGEQ, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.120217387 (260h49m26.991300422s) [noise=true , address=77.90.31.17:4773 , piece_id=SN2X36NEBJIJEDXSLJAROMD3WGOT7DHSC6TGCI4BM7EVRUM777OQ, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:28.723929985 (260h49m28.59501302s) [upload=228688]: Next returned nil segment Dec 16 12:18:28.723956255 (260h49m28.59503929s) [upload=228688]: Waiting for error group managing segments... Dec 16 12:18:29.110968427 (260h49m28.982051472s) [noise=true , address=145.79.227.8:4331 , piece_id=CB7Q2YLIVRLUH5QIFSA3WKICUYRJQVQ5J6SRILARC72SWTMUI7LA, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.112846964 (260h49m28.983929999s) [noise=true , address=145.79.227.41:4086 , piece_id=D2YPUZK4HQENCLATWCFHFX6FOKRDINRTPRERRUVXX63HMZNP5TVA, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.115361984 (260h49m28.986445029s) [noise=true , address=145.79.227.22:4510 , piece_id=JIZVI6JLUC24RSB2FD4AG6DFWY2W243ZDFVEBXAX6WQ77CUBYULA, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.122097569 (260h49m28.993180614s) [noise=true , address=145.79.227.2:4114 , piece_id=QIU7OSO67RMZPWXBCIVCBFVXNBFZ4MMXEUON2EW3U273W6O4XN4Q, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.127395447 (260h49m28.998478482s) [noise=true , address=145.79.227.28:4726 , piece_id=SUMLGDF2DGXG2MXVBYRMGBW6O2V676MQFGATY6RMPAKKHVVELYHA, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.131068261 (260h49m29.002151306s) [noise=true , address=145.79.227.20:4439 , piece_id=UMDFN6JIAD7FPWBZJKXGWCOOTHR7OHHZNUBXVG5H3Q4X2FG6FAWQ, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.179622336 (260h49m29.050705371s) [noise=true , address=145.79.227.4:4185 , piece_id=575AA6RIXXODXK2OVQGSSMD4PWPFOXW6J6MSKIQBK33QHIONZ7XA, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.198721144 (260h49m29.069804179s) [noise=true , address=145.79.227.38:4978 , piece_id=3JYB42IYDMMSEFRJFAIOWPCV5JJAQOILCEGLHMSBWSXT6GPFNWSA, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.203940152 (260h49m29.075023197s) [noise=true , address=145.79.227.3:4150 , piece_id=WXZYNAKPIHDYYMWUZKBI4G7PP4EV2Y6XDQSADOJZASEBMXL7FXQA, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.347406548 (260h49m29.218489583s) [noise=true , address=77.90.31.17:4773 , piece_id=SN2X36NEBJIJEDXSLJAROMD3WGOT7DHSC6TGCI4BM7EVRUM777OQ, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.361791931 (260h49m29.232874966s) [noise=true , address=45.91.116.9:4264 , piece_id=AYTPESZCOSQFBGMYUSHEQC7YEWWZFZFAP6D4F2Y4ITRHGGSIOHAQ, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.447032213 (260h49m29.318115248s) [noise=true , address=45.91.116.5:4297 , piece_id=GRQJKJS4EIM7A5ATMYUSOIFYZTBBKYO72IITZLO72JDM3BKINXQQ, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.932181896 (260h49m29.803264931s) [noise=true , address=77.90.31.13:4629 , piece_id=QV4CXCCUBF6COTIJRYMXMDN2U2MXBHKKFABXAM5C6AZV3XLRV7JQ, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.932199990 (260h49m29.803283025s) [seg_pos={0 0}, upload=228688]: Segment reached optimal threshold of 13 pieces. Dec 16 12:18:29.932354895 (260h49m29.803437931s) [noise=true , address=45.91.116.6:4975 , piece_id=DNIYXILY5EQJVYJSQUPYSCFJPM7UWDRWEW37ES2FGJTIVIFHQALA, seg_pos={0 0}, upload=228688]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 16 12:18:29.932360746 (260h49m29.803443782s) [noise=true , address=45.91.116.6:4975 , piece_id=DNIYXILY5EQJVYJSQUPYSCFJPM7UWDRWEW37ES2FGJTIVIFHQALA, seg_pos={0 0}, upload=228688]: Long tail cancellation due to optimal threshold Dec 16 12:18:29.932609250 (260h49m29.803692295s) [noise=true , address=77.90.31.14:4664 , piece_id=OLEOUQUB6UYPXDQDFR45ZUXDCISMBCZFEZY7CEYLGW4ISZFKLRAQ, seg_pos={0 0}, upload=228688]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9): protocol: read tcp 188.166.225.170:44270->77.90.31.14:4664: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 16 12:18:29.932616132 (260h49m29.803699167s) [noise=true , address=77.90.31.14:4664 , piece_id=OLEOUQUB6UYPXDQDFR45ZUXDCISMBCZFEZY7CEYLGW4ISZFKLRAQ, seg_pos={0 0}, upload=228688]: Long tail cancellation due to optimal threshold Dec 16 12:18:29.932632554 (260h49m29.803715589s) [noise=true , address=77.90.31.19:4842 , piece_id=ZRYPL7FZVCKKNPRLJKV4FIONAKEK7HUDUFRIGF42TVTOHH7KLGEQ, seg_pos={0 0}, upload=228688]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Dec 16 12:18:29.932650127 (260h49m29.803733172s) [noise=true , address=77.90.31.19:4842 , piece_id=ZRYPL7FZVCKKNPRLJKV4FIONAKEK7HUDUFRIGF42TVTOHH7KLGEQ, seg_pos={0 0}, upload=228688]: Long tail cancellation due to optimal threshold Dec 16 12:18:29.932678501 (260h49m29.803761546s) [seg_pos={0 0}, upload=228688]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 12:18:29.932783451 (260h49m29.803866496s) [upload=228688]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 12:18:29.932789533 (260h49m29.803872578s) [upload=228688]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 12:18:30.222615735 (260h49m30.09369879s) [upload=228688]: Done uploading segments... Dec 16 12:18:30.222639069 (260h49m30.093722344s) [upload=228688]: Upload finished. err: <nil> Duration: 6.13628899s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 14 out of 60 is failed (threshold=6) [.....XX.........X....X.XX.......X....X...XXX......X.X...X...] duration: 3.81344067s trace: https://jaeger.dev.storj.io/trace/531b60cce851d866 FAIL (2025-12-16 12:29:32 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.813463814s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 14 out of 60 is failed (threshold=6) [......XX.........X....X.XX.......X....X...XXX......X.X...X..] duration: 3.69903928s trace: https://jaeger.dev.storj.io/trace/531b60cce851d866 FAIL (2025-12-16 12:29:11 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.69908125s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................................XX................] duration: 5.781540044s trace: https://jaeger.dev.storj.io/trace/61b21d2c05e36365 FAIL (2025-12-16 12:28:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 16 12:18:26.867577534 (260h49m26.73866058s) [upload=228688]: Starting upload Dec 16 12:18:26.867632178 (260h49m26.738715213s) [upload=228688]: Done starting upload Dec 16 12:18:26.867824475 (260h49m26.73890751s) [upload=228688]: Uploading segments... Dec 16 12:18:26.868036498 (260h49m26.739119533s) [upload=228688]: Got next segment. Inline: false Dec 16 12:18:26.868072116 (260h49m26.739155151s) [upload=228688]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 12:18:26.868076705 (260h49m26.73915975s) [upload=228688]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 12:18:27.112751871 (260h49m26.983834906s) [seg_pos={0 0}, upload=228688]: Begin upload segment... Dec 16 12:18:27.112835951 (260h49m26.983918996s) [seg_pos={0 0}, upload=228688]: Done begin upload segment. Dec 16 12:18:27.112906205 (260h49m26.98398925s) [noise=true , address=145.79.227.38:4978 , piece_id=3JYB42IYDMMSEFRJFAIOWPCV5JJAQOILCEGLHMSBWSXT6GPFNWSA, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.113024209 (260h49m26.984107255s) [noise=true , address=45.91.116.6:4975 , piece_id=DNIYXILY5EQJVYJSQUPYSCFJPM7UWDRWEW37ES2FGJTIVIFHQALA, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.113642677 (260h49m26.984725722s) [noise=true , address=145.79.227.41:4086 , piece_id=D2YPUZK4HQENCLATWCFHFX6FOKRDINRTPRERRUVXX63HMZNP5TVA, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.114458290 (260h49m26.985541335s) [noise=true , address=145.79.227.3:4150 , piece_id=WXZYNAKPIHDYYMWUZKBI4G7PP4EV2Y6XDQSADOJZASEBMXL7FXQA, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.114469862 (260h49m26.985552907s) [noise=true , address=45.91.116.5:4297 , piece_id=GRQJKJS4EIM7A5ATMYUSOIFYZTBBKYO72IITZLO72JDM3BKINXQQ, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.114980304 (260h49m26.986063339s) [noise=true , address=145.79.227.4:4185 , piece_id=575AA6RIXXODXK2OVQGSSMD4PWPFOXW6J6MSKIQBK33QHIONZ7XA, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.115007396 (260h49m26.986090431s) [noise=true , address=145.79.227.22:4510 , piece_id=JIZVI6JLUC24RSB2FD4AG6DFWY2W243ZDFVEBXAX6WQ77CUBYULA, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.115371549 (260h49m26.986454594s) [noise=true , address=145.79.227.2:4114 , piece_id=QIU7OSO67RMZPWXBCIVCBFVXNBFZ4MMXEUON2EW3U273W6O4XN4Q, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.115802831 (260h49m26.986885866s) [noise=true , address=77.90.31.13:4629 , piece_id=QV4CXCCUBF6COTIJRYMXMDN2U2MXBHKKFABXAM5C6AZV3XLRV7JQ, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.116457888 (260h49m26.987540923s) [noise=true , address=145.79.227.8:4331 , piece_id=CB7Q2YLIVRLUH5QIFSA3WKICUYRJQVQ5J6SRILARC72SWTMUI7LA, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.116866826 (260h49m26.987949861s) [noise=true , address=145.79.227.28:4726 , piece_id=SUMLGDF2DGXG2MXVBYRMGBW6O2V676MQFGATY6RMPAKKHVVELYHA, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.117376858 (260h49m26.988459893s) [noise=true , address=45.91.116.9:4264 , piece_id=AYTPESZCOSQFBGMYUSHEQC7YEWWZFZFAP6D4F2Y4ITRHGGSIOHAQ, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.117883373 (260h49m26.988966408s) [noise=true , address=145.79.227.20:4439 , piece_id=UMDFN6JIAD7FPWBZJKXGWCOOTHR7OHHZNUBXVG5H3Q4X2FG6FAWQ, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.118506488 (260h49m26.989589533s) [noise=true , address=77.90.31.14:4664 , piece_id=OLEOUQUB6UYPXDQDFR45ZUXDCISMBCZFEZY7CEYLGW4ISZFKLRAQ, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.119093947 (260h49m26.990176992s) [noise=true , address=77.90.31.19:4842 , piece_id=ZRYPL7FZVCKKNPRLJKV4FIONAKEK7HUDUFRIGF42TVTOHH7KLGEQ, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.120217387 (260h49m26.991300422s) [noise=true , address=77.90.31.17:4773 , piece_id=SN2X36NEBJIJEDXSLJAROMD3WGOT7DHSC6TGCI4BM7EVRUM777OQ, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:28.723929985 (260h49m28.59501302s) [upload=228688]: Next returned nil segment Dec 16 12:18:28.723956255 (260h49m28.59503929s) [upload=228688]: Waiting for error group managing segments... Dec 16 12:18:29.110968427 (260h49m28.982051472s) [noise=true , address=145.79.227.8:4331 , piece_id=CB7Q2YLIVRLUH5QIFSA3WKICUYRJQVQ5J6SRILARC72SWTMUI7LA, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.112846964 (260h49m28.983929999s) [noise=true , address=145.79.227.41:4086 , piece_id=D2YPUZK4HQENCLATWCFHFX6FOKRDINRTPRERRUVXX63HMZNP5TVA, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.115361984 (260h49m28.986445029s) [noise=true , address=145.79.227.22:4510 , piece_id=JIZVI6JLUC24RSB2FD4AG6DFWY2W243ZDFVEBXAX6WQ77CUBYULA, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.122097569 (260h49m28.993180614s) [noise=true , address=145.79.227.2:4114 , piece_id=QIU7OSO67RMZPWXBCIVCBFVXNBFZ4MMXEUON2EW3U273W6O4XN4Q, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.127395447 (260h49m28.998478482s) [noise=true , address=145.79.227.28:4726 , piece_id=SUMLGDF2DGXG2MXVBYRMGBW6O2V676MQFGATY6RMPAKKHVVELYHA, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.131068261 (260h49m29.002151306s) [noise=true , address=145.79.227.20:4439 , piece_id=UMDFN6JIAD7FPWBZJKXGWCOOTHR7OHHZNUBXVG5H3Q4X2FG6FAWQ, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.179622336 (260h49m29.050705371s) [noise=true , address=145.79.227.4:4185 , piece_id=575AA6RIXXODXK2OVQGSSMD4PWPFOXW6J6MSKIQBK33QHIONZ7XA, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.198721144 (260h49m29.069804179s) [noise=true , address=145.79.227.38:4978 , piece_id=3JYB42IYDMMSEFRJFAIOWPCV5JJAQOILCEGLHMSBWSXT6GPFNWSA, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.203940152 (260h49m29.075023197s) [noise=true , address=145.79.227.3:4150 , piece_id=WXZYNAKPIHDYYMWUZKBI4G7PP4EV2Y6XDQSADOJZASEBMXL7FXQA, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.347406548 (260h49m29.218489583s) [noise=true , address=77.90.31.17:4773 , piece_id=SN2X36NEBJIJEDXSLJAROMD3WGOT7DHSC6TGCI4BM7EVRUM777OQ, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.361791931 (260h49m29.232874966s) [noise=true , address=45.91.116.9:4264 , piece_id=AYTPESZCOSQFBGMYUSHEQC7YEWWZFZFAP6D4F2Y4ITRHGGSIOHAQ, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.447032213 (260h49m29.318115248s) [noise=true , address=45.91.116.5:4297 , piece_id=GRQJKJS4EIM7A5ATMYUSOIFYZTBBKYO72IITZLO72JDM3BKINXQQ, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.932181896 (260h49m29.803264931s) [noise=true , address=77.90.31.13:4629 , piece_id=QV4CXCCUBF6COTIJRYMXMDN2U2MXBHKKFABXAM5C6AZV3XLRV7JQ, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.932199990 (260h49m29.803283025s) [seg_pos={0 0}, upload=228688]: Segment reached optimal threshold of 13 pieces. Dec 16 12:18:29.932354895 (260h49m29.803437931s) [noise=true , address=45.91.116.6:4975 , piece_id=DNIYXILY5EQJVYJSQUPYSCFJPM7UWDRWEW37ES2FGJTIVIFHQALA, seg_pos={0 0}, upload=228688]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 16 12:18:29.932360746 (260h49m29.803443782s) [noise=true , address=45.91.116.6:4975 , piece_id=DNIYXILY5EQJVYJSQUPYSCFJPM7UWDRWEW37ES2FGJTIVIFHQALA, seg_pos={0 0}, upload=228688]: Long tail cancellation due to optimal threshold Dec 16 12:18:29.932609250 (260h49m29.803692295s) [noise=true , address=77.90.31.14:4664 , piece_id=OLEOUQUB6UYPXDQDFR45ZUXDCISMBCZFEZY7CEYLGW4ISZFKLRAQ, seg_pos={0 0}, upload=228688]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9): protocol: read tcp 188.166.225.170:44270->77.90.31.14:4664: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 16 12:18:29.932616132 (260h49m29.803699167s) [noise=true , address=77.90.31.14:4664 , piece_id=OLEOUQUB6UYPXDQDFR45ZUXDCISMBCZFEZY7CEYLGW4ISZFKLRAQ, seg_pos={0 0}, upload=228688]: Long tail cancellation due to optimal threshold Dec 16 12:18:29.932632554 (260h49m29.803715589s) [noise=true , address=77.90.31.19:4842 , piece_id=ZRYPL7FZVCKKNPRLJKV4FIONAKEK7HUDUFRIGF42TVTOHH7KLGEQ, seg_pos={0 0}, upload=228688]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Dec 16 12:18:29.932650127 (260h49m29.803733172s) [noise=true , address=77.90.31.19:4842 , piece_id=ZRYPL7FZVCKKNPRLJKV4FIONAKEK7HUDUFRIGF42TVTOHH7KLGEQ, seg_pos={0 0}, upload=228688]: Long tail cancellation due to optimal threshold Dec 16 12:18:29.932678501 (260h49m29.803761546s) [seg_pos={0 0}, upload=228688]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 12:18:29.932783451 (260h49m29.803866496s) [upload=228688]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 12:18:29.932789533 (260h49m29.803872578s) [upload=228688]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 12:18:30.222615735 (260h49m30.09369879s) [upload=228688]: Done uploading segments... Dec 16 12:18:30.222639069 (260h49m30.093722344s) [upload=228688]: Upload finished. err: <nil> Duration: 5.781563849s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 15 out of 60 is failed (threshold=6) [X......XX.........X....X.XX.......X....X...XXX......X.X...X.] duration: 3.535117914s trace: https://jaeger.dev.storj.io/trace/531b60cce851d866 FAIL (2025-12-16 12:28:41 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.535146136s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........................................XX...............] duration: 5.989324603s trace: https://jaeger.dev.storj.io/trace/61b21d2c05e36365 FAIL (2025-12-16 12:27:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 16 12:18:26.867577534 (260h49m26.73866058s) [upload=228688]: Starting upload Dec 16 12:18:26.867632178 (260h49m26.738715213s) [upload=228688]: Done starting upload Dec 16 12:18:26.867824475 (260h49m26.73890751s) [upload=228688]: Uploading segments... Dec 16 12:18:26.868036498 (260h49m26.739119533s) [upload=228688]: Got next segment. Inline: false Dec 16 12:18:26.868072116 (260h49m26.739155151s) [upload=228688]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 12:18:26.868076705 (260h49m26.73915975s) [upload=228688]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 12:18:27.112751871 (260h49m26.983834906s) [seg_pos={0 0}, upload=228688]: Begin upload segment... Dec 16 12:18:27.112835951 (260h49m26.983918996s) [seg_pos={0 0}, upload=228688]: Done begin upload segment. Dec 16 12:18:27.112906205 (260h49m26.98398925s) [noise=true , address=145.79.227.38:4978 , piece_id=3JYB42IYDMMSEFRJFAIOWPCV5JJAQOILCEGLHMSBWSXT6GPFNWSA, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.113024209 (260h49m26.984107255s) [noise=true , address=45.91.116.6:4975 , piece_id=DNIYXILY5EQJVYJSQUPYSCFJPM7UWDRWEW37ES2FGJTIVIFHQALA, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.113642677 (260h49m26.984725722s) [noise=true , address=145.79.227.41:4086 , piece_id=D2YPUZK4HQENCLATWCFHFX6FOKRDINRTPRERRUVXX63HMZNP5TVA, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.114458290 (260h49m26.985541335s) [noise=true , address=145.79.227.3:4150 , piece_id=WXZYNAKPIHDYYMWUZKBI4G7PP4EV2Y6XDQSADOJZASEBMXL7FXQA, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.114469862 (260h49m26.985552907s) [noise=true , address=45.91.116.5:4297 , piece_id=GRQJKJS4EIM7A5ATMYUSOIFYZTBBKYO72IITZLO72JDM3BKINXQQ, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.114980304 (260h49m26.986063339s) [noise=true , address=145.79.227.4:4185 , piece_id=575AA6RIXXODXK2OVQGSSMD4PWPFOXW6J6MSKIQBK33QHIONZ7XA, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.115007396 (260h49m26.986090431s) [noise=true , address=145.79.227.22:4510 , piece_id=JIZVI6JLUC24RSB2FD4AG6DFWY2W243ZDFVEBXAX6WQ77CUBYULA, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.115371549 (260h49m26.986454594s) [noise=true , address=145.79.227.2:4114 , piece_id=QIU7OSO67RMZPWXBCIVCBFVXNBFZ4MMXEUON2EW3U273W6O4XN4Q, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.115802831 (260h49m26.986885866s) [noise=true , address=77.90.31.13:4629 , piece_id=QV4CXCCUBF6COTIJRYMXMDN2U2MXBHKKFABXAM5C6AZV3XLRV7JQ, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.116457888 (260h49m26.987540923s) [noise=true , address=145.79.227.8:4331 , piece_id=CB7Q2YLIVRLUH5QIFSA3WKICUYRJQVQ5J6SRILARC72SWTMUI7LA, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.116866826 (260h49m26.987949861s) [noise=true , address=145.79.227.28:4726 , piece_id=SUMLGDF2DGXG2MXVBYRMGBW6O2V676MQFGATY6RMPAKKHVVELYHA, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.117376858 (260h49m26.988459893s) [noise=true , address=45.91.116.9:4264 , piece_id=AYTPESZCOSQFBGMYUSHEQC7YEWWZFZFAP6D4F2Y4ITRHGGSIOHAQ, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.117883373 (260h49m26.988966408s) [noise=true , address=145.79.227.20:4439 , piece_id=UMDFN6JIAD7FPWBZJKXGWCOOTHR7OHHZNUBXVG5H3Q4X2FG6FAWQ, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.118506488 (260h49m26.989589533s) [noise=true , address=77.90.31.14:4664 , piece_id=OLEOUQUB6UYPXDQDFR45ZUXDCISMBCZFEZY7CEYLGW4ISZFKLRAQ, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.119093947 (260h49m26.990176992s) [noise=true , address=77.90.31.19:4842 , piece_id=ZRYPL7FZVCKKNPRLJKV4FIONAKEK7HUDUFRIGF42TVTOHH7KLGEQ, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:27.120217387 (260h49m26.991300422s) [noise=true , address=77.90.31.17:4773 , piece_id=SN2X36NEBJIJEDXSLJAROMD3WGOT7DHSC6TGCI4BM7EVRUM777OQ, seg_pos={0 0}, upload=228688]: Uploading piece... Dec 16 12:18:28.723929985 (260h49m28.59501302s) [upload=228688]: Next returned nil segment Dec 16 12:18:28.723956255 (260h49m28.59503929s) [upload=228688]: Waiting for error group managing segments... Dec 16 12:18:29.110968427 (260h49m28.982051472s) [noise=true , address=145.79.227.8:4331 , piece_id=CB7Q2YLIVRLUH5QIFSA3WKICUYRJQVQ5J6SRILARC72SWTMUI7LA, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.112846964 (260h49m28.983929999s) [noise=true , address=145.79.227.41:4086 , piece_id=D2YPUZK4HQENCLATWCFHFX6FOKRDINRTPRERRUVXX63HMZNP5TVA, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.115361984 (260h49m28.986445029s) [noise=true , address=145.79.227.22:4510 , piece_id=JIZVI6JLUC24RSB2FD4AG6DFWY2W243ZDFVEBXAX6WQ77CUBYULA, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.122097569 (260h49m28.993180614s) [noise=true , address=145.79.227.2:4114 , piece_id=QIU7OSO67RMZPWXBCIVCBFVXNBFZ4MMXEUON2EW3U273W6O4XN4Q, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.127395447 (260h49m28.998478482s) [noise=true , address=145.79.227.28:4726 , piece_id=SUMLGDF2DGXG2MXVBYRMGBW6O2V676MQFGATY6RMPAKKHVVELYHA, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.131068261 (260h49m29.002151306s) [noise=true , address=145.79.227.20:4439 , piece_id=UMDFN6JIAD7FPWBZJKXGWCOOTHR7OHHZNUBXVG5H3Q4X2FG6FAWQ, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.179622336 (260h49m29.050705371s) [noise=true , address=145.79.227.4:4185 , piece_id=575AA6RIXXODXK2OVQGSSMD4PWPFOXW6J6MSKIQBK33QHIONZ7XA, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.198721144 (260h49m29.069804179s) [noise=true , address=145.79.227.38:4978 , piece_id=3JYB42IYDMMSEFRJFAIOWPCV5JJAQOILCEGLHMSBWSXT6GPFNWSA, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.203940152 (260h49m29.075023197s) [noise=true , address=145.79.227.3:4150 , piece_id=WXZYNAKPIHDYYMWUZKBI4G7PP4EV2Y6XDQSADOJZASEBMXL7FXQA, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.347406548 (260h49m29.218489583s) [noise=true , address=77.90.31.17:4773 , piece_id=SN2X36NEBJIJEDXSLJAROMD3WGOT7DHSC6TGCI4BM7EVRUM777OQ, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.361791931 (260h49m29.232874966s) [noise=true , address=45.91.116.9:4264 , piece_id=AYTPESZCOSQFBGMYUSHEQC7YEWWZFZFAP6D4F2Y4ITRHGGSIOHAQ, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.447032213 (260h49m29.318115248s) [noise=true , address=45.91.116.5:4297 , piece_id=GRQJKJS4EIM7A5ATMYUSOIFYZTBBKYO72IITZLO72JDM3BKINXQQ, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.932181896 (260h49m29.803264931s) [noise=true , address=77.90.31.13:4629 , piece_id=QV4CXCCUBF6COTIJRYMXMDN2U2MXBHKKFABXAM5C6AZV3XLRV7JQ, seg_pos={0 0}, upload=228688]: Done uploading piece. err: <nil> Dec 16 12:18:29.932199990 (260h49m29.803283025s) [seg_pos={0 0}, upload=228688]: Segment reached optimal threshold of 13 pieces. Dec 16 12:18:29.932354895 (260h49m29.803437931s) [noise=true , address=45.91.116.6:4975 , piece_id=DNIYXILY5EQJVYJSQUPYSCFJPM7UWDRWEW37ES2FGJTIVIFHQALA, seg_pos={0 0}, upload=228688]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 16 12:18:29.932360746 (260h49m29.803443782s) [noise=true , address=45.91.116.6:4975 , piece_id=DNIYXILY5EQJVYJSQUPYSCFJPM7UWDRWEW37ES2FGJTIVIFHQALA, seg_pos={0 0}, upload=228688]: Long tail cancellation due to optimal threshold Dec 16 12:18:29.932609250 (260h49m29.803692295s) [noise=true , address=77.90.31.14:4664 , piece_id=OLEOUQUB6UYPXDQDFR45ZUXDCISMBCZFEZY7CEYLGW4ISZFKLRAQ, seg_pos={0 0}, upload=228688]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9): protocol: read tcp 188.166.225.170:44270->77.90.31.14:4664: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 16 12:18:29.932616132 (260h49m29.803699167s) [noise=true , address=77.90.31.14:4664 , piece_id=OLEOUQUB6UYPXDQDFR45ZUXDCISMBCZFEZY7CEYLGW4ISZFKLRAQ, seg_pos={0 0}, upload=228688]: Long tail cancellation due to optimal threshold Dec 16 12:18:29.932632554 (260h49m29.803715589s) [noise=true , address=77.90.31.19:4842 , piece_id=ZRYPL7FZVCKKNPRLJKV4FIONAKEK7HUDUFRIGF42TVTOHH7KLGEQ, seg_pos={0 0}, upload=228688]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Dec 16 12:18:29.932650127 (260h49m29.803733172s) [noise=true , address=77.90.31.19:4842 , piece_id=ZRYPL7FZVCKKNPRLJKV4FIONAKEK7HUDUFRIGF42TVTOHH7KLGEQ, seg_pos={0 0}, upload=228688]: Long tail cancellation due to optimal threshold Dec 16 12:18:29.932678501 (260h49m29.803761546s) [seg_pos={0 0}, upload=228688]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 12:18:29.932783451 (260h49m29.803866496s) [upload=228688]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 12:18:29.932789533 (260h49m29.803872578s) [upload=228688]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 12:18:30.222615735 (260h49m30.09369879s) [upload=228688]: Done uploading segments... Dec 16 12:18:30.222639069 (260h49m30.093722344s) [upload=228688]: Upload finished. err: <nil> Duration: 5.989352466s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)