Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [X............X..............................................] duration: 3.610231647s trace: https://jaeger.dev.storj.io/trace/772f273c480c0b2b FAIL (2025-12-23 14:49:46 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.610323403s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.........X.XX..X............XX..X.............X.............] duration: 15.873776447s trace: https://jaeger.dev.storj.io/trace/74788274d19d7a46 FAIL (2025-12-23 14:49:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA, Piece ID: PVTAKKVVQ427X5LMVEEJQVOO22NKBC6YRALCX3FYXOFJIBZVS3HQ) context deadline exceeded; piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: 2QPC7ELY3YBFYD7VUV7RHIPFY5PICWG5ZQMLSZRZKSD43Y6LBBOA) context deadline exceeded ======LOGS====== Dec 23 14:40:11.868119876 (431h11m11.739202911s) [upload=378516]: Starting upload Dec 23 14:40:11.868207653 (431h11m11.739290718s) [upload=378516]: Done starting upload Dec 23 14:40:11.868374521 (431h11m11.739457566s) [upload=378516]: Uploading segments... Dec 23 14:40:11.868506252 (431h11m11.739589297s) [upload=378516]: Got next segment. Inline: false Dec 23 14:40:11.868563651 (431h11m11.739646696s) [upload=378516]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 14:40:11.868569853 (431h11m11.739652898s) [upload=378516]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 14:40:12.111082779 (431h11m11.982165825s) [seg_pos={0 0}, upload=378516]: Begin upload segment... Dec 23 14:40:12.111134018 (431h11m11.982217053s) [seg_pos={0 0}, upload=378516]: Done begin upload segment. Dec 23 14:40:12.111151831 (431h11m11.982234866s) [noise=true , address=145.79.227.41:4085 , piece_id=PJ5H7E2LNKRAEAOFEWIVEV4USLM7DJ4XYD4GSQT5N5SQPWKLRLXA, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.111205434 (431h11m11.982288479s) [noise=true , address=77.90.31.15:4697 , piece_id=LRYC2NIG3MUVOMATXPEG4QMMIWKVED3HCBOQ6C7CXAJCSWVXQVTA, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.111360929 (431h11m11.982443974s) [noise=true , address=145.79.227.39:4013 , piece_id=ENCWF72DYPR54VZN4KXDT3U2CS54BJM2BQQ7BYA5ZP373FDWXZVA, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.111802781 (431h11m11.982885826s) [noise=true , address=77.90.31.13:4627 , piece_id=2QPC7ELY3YBFYD7VUV7RHIPFY5PICWG5ZQMLSZRZKSD43Y6LBBOA, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.112248239 (431h11m11.983331294s) [noise=true , address=45.91.116.13:4155 , piece_id=BJIO3BI6TOOJLAO7UWOWDEWS2YA7PHTD6XAQZBJ4R344R5FHXIQA, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.112648541 (431h11m11.983731586s) [noise=true , address=145.79.227.42:4123 , piece_id=P53WTGLAZA2SBX7MEIMYJABAI76T552NUANBWBNXCDAUNR426CSQ, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.113366068 (431h11m11.984449113s) [noise=true , address=45.91.116.5:4298 , piece_id=LNIEPYC6VOANJJAHWGIR54C2NSQNVBLW56WLJS3ATE3RMQI2WTQQ, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.113765128 (431h11m11.984848163s) [noise=true , address=45.91.116.9:4261 , piece_id=PZIQACPNQHHQD4MILGT6KPOOKU4D6O5FPG4D4TWQOVYLCGOPPAGQ, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.114301239 (431h11m11.985384304s) [noise=true , address=145.79.227.37:4565 , piece_id=K62BRPWTKPSD4IJD2NS2P7QC4JZ3QNLH42GO7XZKI2EAXF62K4KQ, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.114637028 (431h11m11.985720063s) [noise=true , address=45.91.116.12:4228 , piece_id=PVTAKKVVQ427X5LMVEEJQVOO22NKBC6YRALCX3FYXOFJIBZVS3HQ, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.115428274 (431h11m11.986511319s) [noise=true , address=145.79.227.49:4339 , piece_id=PQFUWEUSC7MOP6NNXIOE5M4DFA5OIVRLI3NW2DA6DATVCWQMUGJA, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.115717145 (431h11m11.98680018s) [noise=true , address=145.79.227.4:4187 , piece_id=LIXR5F3EXJTP6ALXC3A3V54IKIAK634JCBRONOCFN6RYI5KGOBHQ, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.116286669 (431h11m11.987369714s) [noise=true , address=145.79.227.50:4373 , piece_id=63RBNMEQ2IDHDV73J4FXZHB37KECBX6SWVNRLM36V5EPWKYTKYMQ, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.116830806 (431h11m11.987913841s) [noise=true , address=145.79.227.14:4839 , piece_id=KKNTSJISRHZKMTMDT75F2QDQF4EV6PTQ46QQTOLWFOGX5E5LDWOQ, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.117007933 (431h11m11.988090968s) [noise=true , address=145.79.227.45:4231 , piece_id=ZLYVEFQLJXGZ4PJ6HRXJZLPQWDWTOH7YAZTH3CCMVXUANT2DVGMA, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.117627111 (431h11m11.988710146s) [noise=true , address=145.79.227.11:4730 , piece_id=M5ZYH4B3UNIY65JJFGGRVXBR2VIOEVEJSK4AFO6SMBEFX2ZXYXYQ, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:13.362711746 (431h11m13.233794791s) [noise=true , address=77.90.31.15:4697 , piece_id=LRYC2NIG3MUVOMATXPEG4QMMIWKVED3HCBOQ6C7CXAJCSWVXQVTA, seg_pos={0 0}, upload=378516]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: storage node overloaded, request limit: 2000 Dec 23 14:40:13.824993954 (431h11m13.696076989s) [upload=378516]: Next returned nil segment Dec 23 14:40:13.825024922 (431h11m13.696107957s) [upload=378516]: Waiting for error group managing segments... Dec 23 14:40:14.102416555 (431h11m13.9734996s) [noise=true , address=145.79.227.41:4085 , piece_id=PJ5H7E2LNKRAEAOFEWIVEV4USLM7DJ4XYD4GSQT5N5SQPWKLRLXA, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.104229709 (431h11m13.975312744s) [noise=true , address=145.79.227.4:4187 , piece_id=LIXR5F3EXJTP6ALXC3A3V54IKIAK634JCBRONOCFN6RYI5KGOBHQ, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.108343191 (431h11m13.979426236s) [noise=true , address=145.79.227.50:4373 , piece_id=63RBNMEQ2IDHDV73J4FXZHB37KECBX6SWVNRLM36V5EPWKYTKYMQ, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.108974293 (431h11m13.980057338s) [noise=true , address=145.79.227.11:4730 , piece_id=M5ZYH4B3UNIY65JJFGGRVXBR2VIOEVEJSK4AFO6SMBEFX2ZXYXYQ, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.115937562 (431h11m13.987020607s) [noise=true , address=145.79.227.37:4565 , piece_id=K62BRPWTKPSD4IJD2NS2P7QC4JZ3QNLH42GO7XZKI2EAXF62K4KQ, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.124740384 (431h11m13.995823419s) [noise=true , address=145.79.227.39:4013 , piece_id=ENCWF72DYPR54VZN4KXDT3U2CS54BJM2BQQ7BYA5ZP373FDWXZVA, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.127790853 (431h11m13.998873898s) [noise=true , address=145.79.227.42:4123 , piece_id=P53WTGLAZA2SBX7MEIMYJABAI76T552NUANBWBNXCDAUNR426CSQ, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.185052439 (431h11m14.056135484s) [noise=true , address=145.79.227.49:4339 , piece_id=PQFUWEUSC7MOP6NNXIOE5M4DFA5OIVRLI3NW2DA6DATVCWQMUGJA, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.190499301 (431h11m14.061582346s) [noise=true , address=145.79.227.45:4231 , piece_id=ZLYVEFQLJXGZ4PJ6HRXJZLPQWDWTOH7YAZTH3CCMVXUANT2DVGMA, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.190648225 (431h11m14.06173127s) [noise=true , address=145.79.227.14:4839 , piece_id=KKNTSJISRHZKMTMDT75F2QDQF4EV6PTQ46QQTOLWFOGX5E5LDWOQ, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.382679737 (431h11m14.253762772s) [noise=true , address=45.91.116.9:4261 , piece_id=PZIQACPNQHHQD4MILGT6KPOOKU4D6O5FPG4D4TWQOVYLCGOPPAGQ, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.487075846 (431h11m14.358158881s) [noise=true , address=77.90.31.13:4627 , piece_id=2QPC7ELY3YBFYD7VUV7RHIPFY5PICWG5ZQMLSZRZKSD43Y6LBBOA, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.500579970 (431h11m14.371663005s) [noise=true , address=45.91.116.12:4228 , piece_id=PVTAKKVVQ427X5LMVEEJQVOO22NKBC6YRALCX3FYXOFJIBZVS3HQ, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.500606741 (431h11m14.371689786s) [seg_pos={0 0}, upload=378516]: Segment reached optimal threshold of 13 pieces. Dec 23 14:40:14.500868379 (431h11m14.371951424s) [noise=true , address=45.91.116.5:4298 , piece_id=LNIEPYC6VOANJJAHWGIR54C2NSQNVBLW56WLJS3ATE3RMQI2WTQQ, seg_pos={0 0}, upload=378516]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: write tcp 188.166.225.170:60010->45.91.116.5:4298: use of closed network connection; write tcp 188.166.225.170:60010->45.91.116.5:4298: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60010->45.91.116.5:4298: use of closed network connection Dec 23 14:40:14.500879801 (431h11m14.371962867s) [noise=true , address=45.91.116.5:4298 , piece_id=LNIEPYC6VOANJJAHWGIR54C2NSQNVBLW56WLJS3ATE3RMQI2WTQQ, seg_pos={0 0}, upload=378516]: Long tail cancellation due to optimal threshold Dec 23 14:40:14.501132863 (431h11m14.372215898s) [noise=true , address=45.91.116.13:4155 , piece_id=BJIO3BI6TOOJLAO7UWOWDEWS2YA7PHTD6XAQZBJ4R344R5FHXIQA, seg_pos={0 0}, upload=378516]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Dec 23 14:40:14.501169122 (431h11m14.372252167s) [noise=true , address=45.91.116.13:4155 , piece_id=BJIO3BI6TOOJLAO7UWOWDEWS2YA7PHTD6XAQZBJ4R344R5FHXIQA, seg_pos={0 0}, upload=378516]: Long tail cancellation due to optimal threshold Dec 23 14:40:14.501199750 (431h11m14.372282786s) [seg_pos={0 0}, upload=378516]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 14:40:14.501347652 (431h11m14.372430707s) [upload=378516]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 14:40:14.501356418 (431h11m14.372439463s) [upload=378516]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 14:40:14.797729085 (431h11m14.66881212s) [upload=378516]: Done uploading segments... Dec 23 14:40:14.797748543 (431h11m14.668831588s) [upload=378516]: Upload finished. err: <nil> Duration: 15.87380442s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.X............X.............................................] duration: 3.533549901s trace: https://jaeger.dev.storj.io/trace/772f273c480c0b2b FAIL (2025-12-23 14:49:03 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.533572484s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [..........X.XX..X............XX..X.............X............] duration: 5.995145239s trace: https://jaeger.dev.storj.io/trace/74788274d19d7a46 FAIL (2025-12-23 14:48:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA, Piece ID: PVTAKKVVQ427X5LMVEEJQVOO22NKBC6YRALCX3FYXOFJIBZVS3HQ) context deadline exceeded; piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: 2QPC7ELY3YBFYD7VUV7RHIPFY5PICWG5ZQMLSZRZKSD43Y6LBBOA) context deadline exceeded ======LOGS====== Dec 23 14:40:11.868119876 (431h11m11.739202911s) [upload=378516]: Starting upload Dec 23 14:40:11.868207653 (431h11m11.739290718s) [upload=378516]: Done starting upload Dec 23 14:40:11.868374521 (431h11m11.739457566s) [upload=378516]: Uploading segments... Dec 23 14:40:11.868506252 (431h11m11.739589297s) [upload=378516]: Got next segment. Inline: false Dec 23 14:40:11.868563651 (431h11m11.739646696s) [upload=378516]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 14:40:11.868569853 (431h11m11.739652898s) [upload=378516]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 14:40:12.111082779 (431h11m11.982165825s) [seg_pos={0 0}, upload=378516]: Begin upload segment... Dec 23 14:40:12.111134018 (431h11m11.982217053s) [seg_pos={0 0}, upload=378516]: Done begin upload segment. Dec 23 14:40:12.111151831 (431h11m11.982234866s) [noise=true , address=145.79.227.41:4085 , piece_id=PJ5H7E2LNKRAEAOFEWIVEV4USLM7DJ4XYD4GSQT5N5SQPWKLRLXA, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.111205434 (431h11m11.982288479s) [noise=true , address=77.90.31.15:4697 , piece_id=LRYC2NIG3MUVOMATXPEG4QMMIWKVED3HCBOQ6C7CXAJCSWVXQVTA, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.111360929 (431h11m11.982443974s) [noise=true , address=145.79.227.39:4013 , piece_id=ENCWF72DYPR54VZN4KXDT3U2CS54BJM2BQQ7BYA5ZP373FDWXZVA, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.111802781 (431h11m11.982885826s) [noise=true , address=77.90.31.13:4627 , piece_id=2QPC7ELY3YBFYD7VUV7RHIPFY5PICWG5ZQMLSZRZKSD43Y6LBBOA, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.112248239 (431h11m11.983331294s) [noise=true , address=45.91.116.13:4155 , piece_id=BJIO3BI6TOOJLAO7UWOWDEWS2YA7PHTD6XAQZBJ4R344R5FHXIQA, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.112648541 (431h11m11.983731586s) [noise=true , address=145.79.227.42:4123 , piece_id=P53WTGLAZA2SBX7MEIMYJABAI76T552NUANBWBNXCDAUNR426CSQ, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.113366068 (431h11m11.984449113s) [noise=true , address=45.91.116.5:4298 , piece_id=LNIEPYC6VOANJJAHWGIR54C2NSQNVBLW56WLJS3ATE3RMQI2WTQQ, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.113765128 (431h11m11.984848163s) [noise=true , address=45.91.116.9:4261 , piece_id=PZIQACPNQHHQD4MILGT6KPOOKU4D6O5FPG4D4TWQOVYLCGOPPAGQ, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.114301239 (431h11m11.985384304s) [noise=true , address=145.79.227.37:4565 , piece_id=K62BRPWTKPSD4IJD2NS2P7QC4JZ3QNLH42GO7XZKI2EAXF62K4KQ, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.114637028 (431h11m11.985720063s) [noise=true , address=45.91.116.12:4228 , piece_id=PVTAKKVVQ427X5LMVEEJQVOO22NKBC6YRALCX3FYXOFJIBZVS3HQ, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.115428274 (431h11m11.986511319s) [noise=true , address=145.79.227.49:4339 , piece_id=PQFUWEUSC7MOP6NNXIOE5M4DFA5OIVRLI3NW2DA6DATVCWQMUGJA, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.115717145 (431h11m11.98680018s) [noise=true , address=145.79.227.4:4187 , piece_id=LIXR5F3EXJTP6ALXC3A3V54IKIAK634JCBRONOCFN6RYI5KGOBHQ, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.116286669 (431h11m11.987369714s) [noise=true , address=145.79.227.50:4373 , piece_id=63RBNMEQ2IDHDV73J4FXZHB37KECBX6SWVNRLM36V5EPWKYTKYMQ, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.116830806 (431h11m11.987913841s) [noise=true , address=145.79.227.14:4839 , piece_id=KKNTSJISRHZKMTMDT75F2QDQF4EV6PTQ46QQTOLWFOGX5E5LDWOQ, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.117007933 (431h11m11.988090968s) [noise=true , address=145.79.227.45:4231 , piece_id=ZLYVEFQLJXGZ4PJ6HRXJZLPQWDWTOH7YAZTH3CCMVXUANT2DVGMA, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.117627111 (431h11m11.988710146s) [noise=true , address=145.79.227.11:4730 , piece_id=M5ZYH4B3UNIY65JJFGGRVXBR2VIOEVEJSK4AFO6SMBEFX2ZXYXYQ, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:13.362711746 (431h11m13.233794791s) [noise=true , address=77.90.31.15:4697 , piece_id=LRYC2NIG3MUVOMATXPEG4QMMIWKVED3HCBOQ6C7CXAJCSWVXQVTA, seg_pos={0 0}, upload=378516]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: storage node overloaded, request limit: 2000 Dec 23 14:40:13.824993954 (431h11m13.696076989s) [upload=378516]: Next returned nil segment Dec 23 14:40:13.825024922 (431h11m13.696107957s) [upload=378516]: Waiting for error group managing segments... Dec 23 14:40:14.102416555 (431h11m13.9734996s) [noise=true , address=145.79.227.41:4085 , piece_id=PJ5H7E2LNKRAEAOFEWIVEV4USLM7DJ4XYD4GSQT5N5SQPWKLRLXA, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.104229709 (431h11m13.975312744s) [noise=true , address=145.79.227.4:4187 , piece_id=LIXR5F3EXJTP6ALXC3A3V54IKIAK634JCBRONOCFN6RYI5KGOBHQ, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.108343191 (431h11m13.979426236s) [noise=true , address=145.79.227.50:4373 , piece_id=63RBNMEQ2IDHDV73J4FXZHB37KECBX6SWVNRLM36V5EPWKYTKYMQ, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.108974293 (431h11m13.980057338s) [noise=true , address=145.79.227.11:4730 , piece_id=M5ZYH4B3UNIY65JJFGGRVXBR2VIOEVEJSK4AFO6SMBEFX2ZXYXYQ, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.115937562 (431h11m13.987020607s) [noise=true , address=145.79.227.37:4565 , piece_id=K62BRPWTKPSD4IJD2NS2P7QC4JZ3QNLH42GO7XZKI2EAXF62K4KQ, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.124740384 (431h11m13.995823419s) [noise=true , address=145.79.227.39:4013 , piece_id=ENCWF72DYPR54VZN4KXDT3U2CS54BJM2BQQ7BYA5ZP373FDWXZVA, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.127790853 (431h11m13.998873898s) [noise=true , address=145.79.227.42:4123 , piece_id=P53WTGLAZA2SBX7MEIMYJABAI76T552NUANBWBNXCDAUNR426CSQ, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.185052439 (431h11m14.056135484s) [noise=true , address=145.79.227.49:4339 , piece_id=PQFUWEUSC7MOP6NNXIOE5M4DFA5OIVRLI3NW2DA6DATVCWQMUGJA, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.190499301 (431h11m14.061582346s) [noise=true , address=145.79.227.45:4231 , piece_id=ZLYVEFQLJXGZ4PJ6HRXJZLPQWDWTOH7YAZTH3CCMVXUANT2DVGMA, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.190648225 (431h11m14.06173127s) [noise=true , address=145.79.227.14:4839 , piece_id=KKNTSJISRHZKMTMDT75F2QDQF4EV6PTQ46QQTOLWFOGX5E5LDWOQ, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.382679737 (431h11m14.253762772s) [noise=true , address=45.91.116.9:4261 , piece_id=PZIQACPNQHHQD4MILGT6KPOOKU4D6O5FPG4D4TWQOVYLCGOPPAGQ, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.487075846 (431h11m14.358158881s) [noise=true , address=77.90.31.13:4627 , piece_id=2QPC7ELY3YBFYD7VUV7RHIPFY5PICWG5ZQMLSZRZKSD43Y6LBBOA, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.500579970 (431h11m14.371663005s) [noise=true , address=45.91.116.12:4228 , piece_id=PVTAKKVVQ427X5LMVEEJQVOO22NKBC6YRALCX3FYXOFJIBZVS3HQ, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.500606741 (431h11m14.371689786s) [seg_pos={0 0}, upload=378516]: Segment reached optimal threshold of 13 pieces. Dec 23 14:40:14.500868379 (431h11m14.371951424s) [noise=true , address=45.91.116.5:4298 , piece_id=LNIEPYC6VOANJJAHWGIR54C2NSQNVBLW56WLJS3ATE3RMQI2WTQQ, seg_pos={0 0}, upload=378516]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: write tcp 188.166.225.170:60010->45.91.116.5:4298: use of closed network connection; write tcp 188.166.225.170:60010->45.91.116.5:4298: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60010->45.91.116.5:4298: use of closed network connection Dec 23 14:40:14.500879801 (431h11m14.371962867s) [noise=true , address=45.91.116.5:4298 , piece_id=LNIEPYC6VOANJJAHWGIR54C2NSQNVBLW56WLJS3ATE3RMQI2WTQQ, seg_pos={0 0}, upload=378516]: Long tail cancellation due to optimal threshold Dec 23 14:40:14.501132863 (431h11m14.372215898s) [noise=true , address=45.91.116.13:4155 , piece_id=BJIO3BI6TOOJLAO7UWOWDEWS2YA7PHTD6XAQZBJ4R344R5FHXIQA, seg_pos={0 0}, upload=378516]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Dec 23 14:40:14.501169122 (431h11m14.372252167s) [noise=true , address=45.91.116.13:4155 , piece_id=BJIO3BI6TOOJLAO7UWOWDEWS2YA7PHTD6XAQZBJ4R344R5FHXIQA, seg_pos={0 0}, upload=378516]: Long tail cancellation due to optimal threshold Dec 23 14:40:14.501199750 (431h11m14.372282786s) [seg_pos={0 0}, upload=378516]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 14:40:14.501347652 (431h11m14.372430707s) [upload=378516]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 14:40:14.501356418 (431h11m14.372439463s) [upload=378516]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 14:40:14.797729085 (431h11m14.66881212s) [upload=378516]: Done uploading segments... Dec 23 14:40:14.797748543 (431h11m14.668831588s) [upload=378516]: Upload finished. err: <nil> Duration: 5.995273202s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..X............X............................................] duration: 3.878809075s trace: https://jaeger.dev.storj.io/trace/772f273c480c0b2b FAIL (2025-12-23 14:48:13 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.878832399s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [...X............X...........................................] duration: 3.570648603s trace: https://jaeger.dev.storj.io/trace/772f273c480c0b2b FAIL (2025-12-23 14:47:39 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.570667148s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [X..........X.XX..X............XX..X.............X...........] duration: 7.090685945s trace: https://jaeger.dev.storj.io/trace/74788274d19d7a46 FAIL (2025-12-23 14:47:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA, Piece ID: PVTAKKVVQ427X5LMVEEJQVOO22NKBC6YRALCX3FYXOFJIBZVS3HQ) context deadline exceeded; piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: 2QPC7ELY3YBFYD7VUV7RHIPFY5PICWG5ZQMLSZRZKSD43Y6LBBOA) context deadline exceeded ======LOGS====== Dec 23 14:40:11.868119876 (431h11m11.739202911s) [upload=378516]: Starting upload Dec 23 14:40:11.868207653 (431h11m11.739290718s) [upload=378516]: Done starting upload Dec 23 14:40:11.868374521 (431h11m11.739457566s) [upload=378516]: Uploading segments... Dec 23 14:40:11.868506252 (431h11m11.739589297s) [upload=378516]: Got next segment. Inline: false Dec 23 14:40:11.868563651 (431h11m11.739646696s) [upload=378516]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 14:40:11.868569853 (431h11m11.739652898s) [upload=378516]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 14:40:12.111082779 (431h11m11.982165825s) [seg_pos={0 0}, upload=378516]: Begin upload segment... Dec 23 14:40:12.111134018 (431h11m11.982217053s) [seg_pos={0 0}, upload=378516]: Done begin upload segment. Dec 23 14:40:12.111151831 (431h11m11.982234866s) [noise=true , address=145.79.227.41:4085 , piece_id=PJ5H7E2LNKRAEAOFEWIVEV4USLM7DJ4XYD4GSQT5N5SQPWKLRLXA, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.111205434 (431h11m11.982288479s) [noise=true , address=77.90.31.15:4697 , piece_id=LRYC2NIG3MUVOMATXPEG4QMMIWKVED3HCBOQ6C7CXAJCSWVXQVTA, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.111360929 (431h11m11.982443974s) [noise=true , address=145.79.227.39:4013 , piece_id=ENCWF72DYPR54VZN4KXDT3U2CS54BJM2BQQ7BYA5ZP373FDWXZVA, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.111802781 (431h11m11.982885826s) [noise=true , address=77.90.31.13:4627 , piece_id=2QPC7ELY3YBFYD7VUV7RHIPFY5PICWG5ZQMLSZRZKSD43Y6LBBOA, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.112248239 (431h11m11.983331294s) [noise=true , address=45.91.116.13:4155 , piece_id=BJIO3BI6TOOJLAO7UWOWDEWS2YA7PHTD6XAQZBJ4R344R5FHXIQA, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.112648541 (431h11m11.983731586s) [noise=true , address=145.79.227.42:4123 , piece_id=P53WTGLAZA2SBX7MEIMYJABAI76T552NUANBWBNXCDAUNR426CSQ, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.113366068 (431h11m11.984449113s) [noise=true , address=45.91.116.5:4298 , piece_id=LNIEPYC6VOANJJAHWGIR54C2NSQNVBLW56WLJS3ATE3RMQI2WTQQ, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.113765128 (431h11m11.984848163s) [noise=true , address=45.91.116.9:4261 , piece_id=PZIQACPNQHHQD4MILGT6KPOOKU4D6O5FPG4D4TWQOVYLCGOPPAGQ, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.114301239 (431h11m11.985384304s) [noise=true , address=145.79.227.37:4565 , piece_id=K62BRPWTKPSD4IJD2NS2P7QC4JZ3QNLH42GO7XZKI2EAXF62K4KQ, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.114637028 (431h11m11.985720063s) [noise=true , address=45.91.116.12:4228 , piece_id=PVTAKKVVQ427X5LMVEEJQVOO22NKBC6YRALCX3FYXOFJIBZVS3HQ, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.115428274 (431h11m11.986511319s) [noise=true , address=145.79.227.49:4339 , piece_id=PQFUWEUSC7MOP6NNXIOE5M4DFA5OIVRLI3NW2DA6DATVCWQMUGJA, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.115717145 (431h11m11.98680018s) [noise=true , address=145.79.227.4:4187 , piece_id=LIXR5F3EXJTP6ALXC3A3V54IKIAK634JCBRONOCFN6RYI5KGOBHQ, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.116286669 (431h11m11.987369714s) [noise=true , address=145.79.227.50:4373 , piece_id=63RBNMEQ2IDHDV73J4FXZHB37KECBX6SWVNRLM36V5EPWKYTKYMQ, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.116830806 (431h11m11.987913841s) [noise=true , address=145.79.227.14:4839 , piece_id=KKNTSJISRHZKMTMDT75F2QDQF4EV6PTQ46QQTOLWFOGX5E5LDWOQ, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.117007933 (431h11m11.988090968s) [noise=true , address=145.79.227.45:4231 , piece_id=ZLYVEFQLJXGZ4PJ6HRXJZLPQWDWTOH7YAZTH3CCMVXUANT2DVGMA, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.117627111 (431h11m11.988710146s) [noise=true , address=145.79.227.11:4730 , piece_id=M5ZYH4B3UNIY65JJFGGRVXBR2VIOEVEJSK4AFO6SMBEFX2ZXYXYQ, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:13.362711746 (431h11m13.233794791s) [noise=true , address=77.90.31.15:4697 , piece_id=LRYC2NIG3MUVOMATXPEG4QMMIWKVED3HCBOQ6C7CXAJCSWVXQVTA, seg_pos={0 0}, upload=378516]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: storage node overloaded, request limit: 2000 Dec 23 14:40:13.824993954 (431h11m13.696076989s) [upload=378516]: Next returned nil segment Dec 23 14:40:13.825024922 (431h11m13.696107957s) [upload=378516]: Waiting for error group managing segments... Dec 23 14:40:14.102416555 (431h11m13.9734996s) [noise=true , address=145.79.227.41:4085 , piece_id=PJ5H7E2LNKRAEAOFEWIVEV4USLM7DJ4XYD4GSQT5N5SQPWKLRLXA, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.104229709 (431h11m13.975312744s) [noise=true , address=145.79.227.4:4187 , piece_id=LIXR5F3EXJTP6ALXC3A3V54IKIAK634JCBRONOCFN6RYI5KGOBHQ, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.108343191 (431h11m13.979426236s) [noise=true , address=145.79.227.50:4373 , piece_id=63RBNMEQ2IDHDV73J4FXZHB37KECBX6SWVNRLM36V5EPWKYTKYMQ, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.108974293 (431h11m13.980057338s) [noise=true , address=145.79.227.11:4730 , piece_id=M5ZYH4B3UNIY65JJFGGRVXBR2VIOEVEJSK4AFO6SMBEFX2ZXYXYQ, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.115937562 (431h11m13.987020607s) [noise=true , address=145.79.227.37:4565 , piece_id=K62BRPWTKPSD4IJD2NS2P7QC4JZ3QNLH42GO7XZKI2EAXF62K4KQ, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.124740384 (431h11m13.995823419s) [noise=true , address=145.79.227.39:4013 , piece_id=ENCWF72DYPR54VZN4KXDT3U2CS54BJM2BQQ7BYA5ZP373FDWXZVA, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.127790853 (431h11m13.998873898s) [noise=true , address=145.79.227.42:4123 , piece_id=P53WTGLAZA2SBX7MEIMYJABAI76T552NUANBWBNXCDAUNR426CSQ, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.185052439 (431h11m14.056135484s) [noise=true , address=145.79.227.49:4339 , piece_id=PQFUWEUSC7MOP6NNXIOE5M4DFA5OIVRLI3NW2DA6DATVCWQMUGJA, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.190499301 (431h11m14.061582346s) [noise=true , address=145.79.227.45:4231 , piece_id=ZLYVEFQLJXGZ4PJ6HRXJZLPQWDWTOH7YAZTH3CCMVXUANT2DVGMA, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.190648225 (431h11m14.06173127s) [noise=true , address=145.79.227.14:4839 , piece_id=KKNTSJISRHZKMTMDT75F2QDQF4EV6PTQ46QQTOLWFOGX5E5LDWOQ, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.382679737 (431h11m14.253762772s) [noise=true , address=45.91.116.9:4261 , piece_id=PZIQACPNQHHQD4MILGT6KPOOKU4D6O5FPG4D4TWQOVYLCGOPPAGQ, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.487075846 (431h11m14.358158881s) [noise=true , address=77.90.31.13:4627 , piece_id=2QPC7ELY3YBFYD7VUV7RHIPFY5PICWG5ZQMLSZRZKSD43Y6LBBOA, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.500579970 (431h11m14.371663005s) [noise=true , address=45.91.116.12:4228 , piece_id=PVTAKKVVQ427X5LMVEEJQVOO22NKBC6YRALCX3FYXOFJIBZVS3HQ, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.500606741 (431h11m14.371689786s) [seg_pos={0 0}, upload=378516]: Segment reached optimal threshold of 13 pieces. Dec 23 14:40:14.500868379 (431h11m14.371951424s) [noise=true , address=45.91.116.5:4298 , piece_id=LNIEPYC6VOANJJAHWGIR54C2NSQNVBLW56WLJS3ATE3RMQI2WTQQ, seg_pos={0 0}, upload=378516]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: write tcp 188.166.225.170:60010->45.91.116.5:4298: use of closed network connection; write tcp 188.166.225.170:60010->45.91.116.5:4298: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60010->45.91.116.5:4298: use of closed network connection Dec 23 14:40:14.500879801 (431h11m14.371962867s) [noise=true , address=45.91.116.5:4298 , piece_id=LNIEPYC6VOANJJAHWGIR54C2NSQNVBLW56WLJS3ATE3RMQI2WTQQ, seg_pos={0 0}, upload=378516]: Long tail cancellation due to optimal threshold Dec 23 14:40:14.501132863 (431h11m14.372215898s) [noise=true , address=45.91.116.13:4155 , piece_id=BJIO3BI6TOOJLAO7UWOWDEWS2YA7PHTD6XAQZBJ4R344R5FHXIQA, seg_pos={0 0}, upload=378516]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Dec 23 14:40:14.501169122 (431h11m14.372252167s) [noise=true , address=45.91.116.13:4155 , piece_id=BJIO3BI6TOOJLAO7UWOWDEWS2YA7PHTD6XAQZBJ4R344R5FHXIQA, seg_pos={0 0}, upload=378516]: Long tail cancellation due to optimal threshold Dec 23 14:40:14.501199750 (431h11m14.372282786s) [seg_pos={0 0}, upload=378516]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 14:40:14.501347652 (431h11m14.372430707s) [upload=378516]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 14:40:14.501356418 (431h11m14.372439463s) [upload=378516]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 14:40:14.797729085 (431h11m14.66881212s) [upload=378516]: Done uploading segments... Dec 23 14:40:14.797748543 (431h11m14.668831588s) [upload=378516]: Upload finished. err: <nil> Duration: 7.090717756s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [.X..........X.XX..X............XX..X.............X..........] duration: 16.815490797s trace: https://jaeger.dev.storj.io/trace/74788274d19d7a46 FAIL (2025-12-23 14:47:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA, Piece ID: PVTAKKVVQ427X5LMVEEJQVOO22NKBC6YRALCX3FYXOFJIBZVS3HQ) context deadline exceeded; piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: 2QPC7ELY3YBFYD7VUV7RHIPFY5PICWG5ZQMLSZRZKSD43Y6LBBOA) context deadline exceeded ======LOGS====== Dec 23 14:40:11.868119876 (431h11m11.739202911s) [upload=378516]: Starting upload Dec 23 14:40:11.868207653 (431h11m11.739290718s) [upload=378516]: Done starting upload Dec 23 14:40:11.868374521 (431h11m11.739457566s) [upload=378516]: Uploading segments... Dec 23 14:40:11.868506252 (431h11m11.739589297s) [upload=378516]: Got next segment. Inline: false Dec 23 14:40:11.868563651 (431h11m11.739646696s) [upload=378516]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 14:40:11.868569853 (431h11m11.739652898s) [upload=378516]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 14:40:12.111082779 (431h11m11.982165825s) [seg_pos={0 0}, upload=378516]: Begin upload segment... Dec 23 14:40:12.111134018 (431h11m11.982217053s) [seg_pos={0 0}, upload=378516]: Done begin upload segment. Dec 23 14:40:12.111151831 (431h11m11.982234866s) [noise=true , address=145.79.227.41:4085 , piece_id=PJ5H7E2LNKRAEAOFEWIVEV4USLM7DJ4XYD4GSQT5N5SQPWKLRLXA, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.111205434 (431h11m11.982288479s) [noise=true , address=77.90.31.15:4697 , piece_id=LRYC2NIG3MUVOMATXPEG4QMMIWKVED3HCBOQ6C7CXAJCSWVXQVTA, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.111360929 (431h11m11.982443974s) [noise=true , address=145.79.227.39:4013 , piece_id=ENCWF72DYPR54VZN4KXDT3U2CS54BJM2BQQ7BYA5ZP373FDWXZVA, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.111802781 (431h11m11.982885826s) [noise=true , address=77.90.31.13:4627 , piece_id=2QPC7ELY3YBFYD7VUV7RHIPFY5PICWG5ZQMLSZRZKSD43Y6LBBOA, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.112248239 (431h11m11.983331294s) [noise=true , address=45.91.116.13:4155 , piece_id=BJIO3BI6TOOJLAO7UWOWDEWS2YA7PHTD6XAQZBJ4R344R5FHXIQA, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.112648541 (431h11m11.983731586s) [noise=true , address=145.79.227.42:4123 , piece_id=P53WTGLAZA2SBX7MEIMYJABAI76T552NUANBWBNXCDAUNR426CSQ, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.113366068 (431h11m11.984449113s) [noise=true , address=45.91.116.5:4298 , piece_id=LNIEPYC6VOANJJAHWGIR54C2NSQNVBLW56WLJS3ATE3RMQI2WTQQ, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.113765128 (431h11m11.984848163s) [noise=true , address=45.91.116.9:4261 , piece_id=PZIQACPNQHHQD4MILGT6KPOOKU4D6O5FPG4D4TWQOVYLCGOPPAGQ, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.114301239 (431h11m11.985384304s) [noise=true , address=145.79.227.37:4565 , piece_id=K62BRPWTKPSD4IJD2NS2P7QC4JZ3QNLH42GO7XZKI2EAXF62K4KQ, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.114637028 (431h11m11.985720063s) [noise=true , address=45.91.116.12:4228 , piece_id=PVTAKKVVQ427X5LMVEEJQVOO22NKBC6YRALCX3FYXOFJIBZVS3HQ, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.115428274 (431h11m11.986511319s) [noise=true , address=145.79.227.49:4339 , piece_id=PQFUWEUSC7MOP6NNXIOE5M4DFA5OIVRLI3NW2DA6DATVCWQMUGJA, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.115717145 (431h11m11.98680018s) [noise=true , address=145.79.227.4:4187 , piece_id=LIXR5F3EXJTP6ALXC3A3V54IKIAK634JCBRONOCFN6RYI5KGOBHQ, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.116286669 (431h11m11.987369714s) [noise=true , address=145.79.227.50:4373 , piece_id=63RBNMEQ2IDHDV73J4FXZHB37KECBX6SWVNRLM36V5EPWKYTKYMQ, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.116830806 (431h11m11.987913841s) [noise=true , address=145.79.227.14:4839 , piece_id=KKNTSJISRHZKMTMDT75F2QDQF4EV6PTQ46QQTOLWFOGX5E5LDWOQ, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.117007933 (431h11m11.988090968s) [noise=true , address=145.79.227.45:4231 , piece_id=ZLYVEFQLJXGZ4PJ6HRXJZLPQWDWTOH7YAZTH3CCMVXUANT2DVGMA, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.117627111 (431h11m11.988710146s) [noise=true , address=145.79.227.11:4730 , piece_id=M5ZYH4B3UNIY65JJFGGRVXBR2VIOEVEJSK4AFO6SMBEFX2ZXYXYQ, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:13.362711746 (431h11m13.233794791s) [noise=true , address=77.90.31.15:4697 , piece_id=LRYC2NIG3MUVOMATXPEG4QMMIWKVED3HCBOQ6C7CXAJCSWVXQVTA, seg_pos={0 0}, upload=378516]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: storage node overloaded, request limit: 2000 Dec 23 14:40:13.824993954 (431h11m13.696076989s) [upload=378516]: Next returned nil segment Dec 23 14:40:13.825024922 (431h11m13.696107957s) [upload=378516]: Waiting for error group managing segments... Dec 23 14:40:14.102416555 (431h11m13.9734996s) [noise=true , address=145.79.227.41:4085 , piece_id=PJ5H7E2LNKRAEAOFEWIVEV4USLM7DJ4XYD4GSQT5N5SQPWKLRLXA, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.104229709 (431h11m13.975312744s) [noise=true , address=145.79.227.4:4187 , piece_id=LIXR5F3EXJTP6ALXC3A3V54IKIAK634JCBRONOCFN6RYI5KGOBHQ, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.108343191 (431h11m13.979426236s) [noise=true , address=145.79.227.50:4373 , piece_id=63RBNMEQ2IDHDV73J4FXZHB37KECBX6SWVNRLM36V5EPWKYTKYMQ, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.108974293 (431h11m13.980057338s) [noise=true , address=145.79.227.11:4730 , piece_id=M5ZYH4B3UNIY65JJFGGRVXBR2VIOEVEJSK4AFO6SMBEFX2ZXYXYQ, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.115937562 (431h11m13.987020607s) [noise=true , address=145.79.227.37:4565 , piece_id=K62BRPWTKPSD4IJD2NS2P7QC4JZ3QNLH42GO7XZKI2EAXF62K4KQ, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.124740384 (431h11m13.995823419s) [noise=true , address=145.79.227.39:4013 , piece_id=ENCWF72DYPR54VZN4KXDT3U2CS54BJM2BQQ7BYA5ZP373FDWXZVA, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.127790853 (431h11m13.998873898s) [noise=true , address=145.79.227.42:4123 , piece_id=P53WTGLAZA2SBX7MEIMYJABAI76T552NUANBWBNXCDAUNR426CSQ, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.185052439 (431h11m14.056135484s) [noise=true , address=145.79.227.49:4339 , piece_id=PQFUWEUSC7MOP6NNXIOE5M4DFA5OIVRLI3NW2DA6DATVCWQMUGJA, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.190499301 (431h11m14.061582346s) [noise=true , address=145.79.227.45:4231 , piece_id=ZLYVEFQLJXGZ4PJ6HRXJZLPQWDWTOH7YAZTH3CCMVXUANT2DVGMA, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.190648225 (431h11m14.06173127s) [noise=true , address=145.79.227.14:4839 , piece_id=KKNTSJISRHZKMTMDT75F2QDQF4EV6PTQ46QQTOLWFOGX5E5LDWOQ, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.382679737 (431h11m14.253762772s) [noise=true , address=45.91.116.9:4261 , piece_id=PZIQACPNQHHQD4MILGT6KPOOKU4D6O5FPG4D4TWQOVYLCGOPPAGQ, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.487075846 (431h11m14.358158881s) [noise=true , address=77.90.31.13:4627 , piece_id=2QPC7ELY3YBFYD7VUV7RHIPFY5PICWG5ZQMLSZRZKSD43Y6LBBOA, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.500579970 (431h11m14.371663005s) [noise=true , address=45.91.116.12:4228 , piece_id=PVTAKKVVQ427X5LMVEEJQVOO22NKBC6YRALCX3FYXOFJIBZVS3HQ, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.500606741 (431h11m14.371689786s) [seg_pos={0 0}, upload=378516]: Segment reached optimal threshold of 13 pieces. Dec 23 14:40:14.500868379 (431h11m14.371951424s) [noise=true , address=45.91.116.5:4298 , piece_id=LNIEPYC6VOANJJAHWGIR54C2NSQNVBLW56WLJS3ATE3RMQI2WTQQ, seg_pos={0 0}, upload=378516]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: write tcp 188.166.225.170:60010->45.91.116.5:4298: use of closed network connection; write tcp 188.166.225.170:60010->45.91.116.5:4298: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60010->45.91.116.5:4298: use of closed network connection Dec 23 14:40:14.500879801 (431h11m14.371962867s) [noise=true , address=45.91.116.5:4298 , piece_id=LNIEPYC6VOANJJAHWGIR54C2NSQNVBLW56WLJS3ATE3RMQI2WTQQ, seg_pos={0 0}, upload=378516]: Long tail cancellation due to optimal threshold Dec 23 14:40:14.501132863 (431h11m14.372215898s) [noise=true , address=45.91.116.13:4155 , piece_id=BJIO3BI6TOOJLAO7UWOWDEWS2YA7PHTD6XAQZBJ4R344R5FHXIQA, seg_pos={0 0}, upload=378516]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Dec 23 14:40:14.501169122 (431h11m14.372252167s) [noise=true , address=45.91.116.13:4155 , piece_id=BJIO3BI6TOOJLAO7UWOWDEWS2YA7PHTD6XAQZBJ4R344R5FHXIQA, seg_pos={0 0}, upload=378516]: Long tail cancellation due to optimal threshold Dec 23 14:40:14.501199750 (431h11m14.372282786s) [seg_pos={0 0}, upload=378516]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 14:40:14.501347652 (431h11m14.372430707s) [upload=378516]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 14:40:14.501356418 (431h11m14.372439463s) [upload=378516]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 14:40:14.797729085 (431h11m14.66881212s) [upload=378516]: Done uploading segments... Dec 23 14:40:14.797748543 (431h11m14.668831588s) [upload=378516]: Upload finished. err: <nil> Duration: 16.815510094s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [....X............X..........................................] duration: 3.680417382s trace: https://jaeger.dev.storj.io/trace/772f273c480c0b2b FAIL (2025-12-23 14:46:58 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.680447279s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [..X..........X.XX..X............XX..X.............X.........] duration: 6.00890938s trace: https://jaeger.dev.storj.io/trace/74788274d19d7a46 FAIL (2025-12-23 14:45:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA, Piece ID: PVTAKKVVQ427X5LMVEEJQVOO22NKBC6YRALCX3FYXOFJIBZVS3HQ) context deadline exceeded; piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: 2QPC7ELY3YBFYD7VUV7RHIPFY5PICWG5ZQMLSZRZKSD43Y6LBBOA) context deadline exceeded ======LOGS====== Dec 23 14:40:11.868119876 (431h11m11.739202911s) [upload=378516]: Starting upload Dec 23 14:40:11.868207653 (431h11m11.739290718s) [upload=378516]: Done starting upload Dec 23 14:40:11.868374521 (431h11m11.739457566s) [upload=378516]: Uploading segments... Dec 23 14:40:11.868506252 (431h11m11.739589297s) [upload=378516]: Got next segment. Inline: false Dec 23 14:40:11.868563651 (431h11m11.739646696s) [upload=378516]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 14:40:11.868569853 (431h11m11.739652898s) [upload=378516]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 14:40:12.111082779 (431h11m11.982165825s) [seg_pos={0 0}, upload=378516]: Begin upload segment... Dec 23 14:40:12.111134018 (431h11m11.982217053s) [seg_pos={0 0}, upload=378516]: Done begin upload segment. Dec 23 14:40:12.111151831 (431h11m11.982234866s) [noise=true , address=145.79.227.41:4085 , piece_id=PJ5H7E2LNKRAEAOFEWIVEV4USLM7DJ4XYD4GSQT5N5SQPWKLRLXA, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.111205434 (431h11m11.982288479s) [noise=true , address=77.90.31.15:4697 , piece_id=LRYC2NIG3MUVOMATXPEG4QMMIWKVED3HCBOQ6C7CXAJCSWVXQVTA, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.111360929 (431h11m11.982443974s) [noise=true , address=145.79.227.39:4013 , piece_id=ENCWF72DYPR54VZN4KXDT3U2CS54BJM2BQQ7BYA5ZP373FDWXZVA, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.111802781 (431h11m11.982885826s) [noise=true , address=77.90.31.13:4627 , piece_id=2QPC7ELY3YBFYD7VUV7RHIPFY5PICWG5ZQMLSZRZKSD43Y6LBBOA, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.112248239 (431h11m11.983331294s) [noise=true , address=45.91.116.13:4155 , piece_id=BJIO3BI6TOOJLAO7UWOWDEWS2YA7PHTD6XAQZBJ4R344R5FHXIQA, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.112648541 (431h11m11.983731586s) [noise=true , address=145.79.227.42:4123 , piece_id=P53WTGLAZA2SBX7MEIMYJABAI76T552NUANBWBNXCDAUNR426CSQ, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.113366068 (431h11m11.984449113s) [noise=true , address=45.91.116.5:4298 , piece_id=LNIEPYC6VOANJJAHWGIR54C2NSQNVBLW56WLJS3ATE3RMQI2WTQQ, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.113765128 (431h11m11.984848163s) [noise=true , address=45.91.116.9:4261 , piece_id=PZIQACPNQHHQD4MILGT6KPOOKU4D6O5FPG4D4TWQOVYLCGOPPAGQ, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.114301239 (431h11m11.985384304s) [noise=true , address=145.79.227.37:4565 , piece_id=K62BRPWTKPSD4IJD2NS2P7QC4JZ3QNLH42GO7XZKI2EAXF62K4KQ, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.114637028 (431h11m11.985720063s) [noise=true , address=45.91.116.12:4228 , piece_id=PVTAKKVVQ427X5LMVEEJQVOO22NKBC6YRALCX3FYXOFJIBZVS3HQ, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.115428274 (431h11m11.986511319s) [noise=true , address=145.79.227.49:4339 , piece_id=PQFUWEUSC7MOP6NNXIOE5M4DFA5OIVRLI3NW2DA6DATVCWQMUGJA, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.115717145 (431h11m11.98680018s) [noise=true , address=145.79.227.4:4187 , piece_id=LIXR5F3EXJTP6ALXC3A3V54IKIAK634JCBRONOCFN6RYI5KGOBHQ, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.116286669 (431h11m11.987369714s) [noise=true , address=145.79.227.50:4373 , piece_id=63RBNMEQ2IDHDV73J4FXZHB37KECBX6SWVNRLM36V5EPWKYTKYMQ, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.116830806 (431h11m11.987913841s) [noise=true , address=145.79.227.14:4839 , piece_id=KKNTSJISRHZKMTMDT75F2QDQF4EV6PTQ46QQTOLWFOGX5E5LDWOQ, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.117007933 (431h11m11.988090968s) [noise=true , address=145.79.227.45:4231 , piece_id=ZLYVEFQLJXGZ4PJ6HRXJZLPQWDWTOH7YAZTH3CCMVXUANT2DVGMA, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.117627111 (431h11m11.988710146s) [noise=true , address=145.79.227.11:4730 , piece_id=M5ZYH4B3UNIY65JJFGGRVXBR2VIOEVEJSK4AFO6SMBEFX2ZXYXYQ, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:13.362711746 (431h11m13.233794791s) [noise=true , address=77.90.31.15:4697 , piece_id=LRYC2NIG3MUVOMATXPEG4QMMIWKVED3HCBOQ6C7CXAJCSWVXQVTA, seg_pos={0 0}, upload=378516]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: storage node overloaded, request limit: 2000 Dec 23 14:40:13.824993954 (431h11m13.696076989s) [upload=378516]: Next returned nil segment Dec 23 14:40:13.825024922 (431h11m13.696107957s) [upload=378516]: Waiting for error group managing segments... Dec 23 14:40:14.102416555 (431h11m13.9734996s) [noise=true , address=145.79.227.41:4085 , piece_id=PJ5H7E2LNKRAEAOFEWIVEV4USLM7DJ4XYD4GSQT5N5SQPWKLRLXA, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.104229709 (431h11m13.975312744s) [noise=true , address=145.79.227.4:4187 , piece_id=LIXR5F3EXJTP6ALXC3A3V54IKIAK634JCBRONOCFN6RYI5KGOBHQ, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.108343191 (431h11m13.979426236s) [noise=true , address=145.79.227.50:4373 , piece_id=63RBNMEQ2IDHDV73J4FXZHB37KECBX6SWVNRLM36V5EPWKYTKYMQ, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.108974293 (431h11m13.980057338s) [noise=true , address=145.79.227.11:4730 , piece_id=M5ZYH4B3UNIY65JJFGGRVXBR2VIOEVEJSK4AFO6SMBEFX2ZXYXYQ, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.115937562 (431h11m13.987020607s) [noise=true , address=145.79.227.37:4565 , piece_id=K62BRPWTKPSD4IJD2NS2P7QC4JZ3QNLH42GO7XZKI2EAXF62K4KQ, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.124740384 (431h11m13.995823419s) [noise=true , address=145.79.227.39:4013 , piece_id=ENCWF72DYPR54VZN4KXDT3U2CS54BJM2BQQ7BYA5ZP373FDWXZVA, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.127790853 (431h11m13.998873898s) [noise=true , address=145.79.227.42:4123 , piece_id=P53WTGLAZA2SBX7MEIMYJABAI76T552NUANBWBNXCDAUNR426CSQ, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.185052439 (431h11m14.056135484s) [noise=true , address=145.79.227.49:4339 , piece_id=PQFUWEUSC7MOP6NNXIOE5M4DFA5OIVRLI3NW2DA6DATVCWQMUGJA, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.190499301 (431h11m14.061582346s) [noise=true , address=145.79.227.45:4231 , piece_id=ZLYVEFQLJXGZ4PJ6HRXJZLPQWDWTOH7YAZTH3CCMVXUANT2DVGMA, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.190648225 (431h11m14.06173127s) [noise=true , address=145.79.227.14:4839 , piece_id=KKNTSJISRHZKMTMDT75F2QDQF4EV6PTQ46QQTOLWFOGX5E5LDWOQ, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.382679737 (431h11m14.253762772s) [noise=true , address=45.91.116.9:4261 , piece_id=PZIQACPNQHHQD4MILGT6KPOOKU4D6O5FPG4D4TWQOVYLCGOPPAGQ, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.487075846 (431h11m14.358158881s) [noise=true , address=77.90.31.13:4627 , piece_id=2QPC7ELY3YBFYD7VUV7RHIPFY5PICWG5ZQMLSZRZKSD43Y6LBBOA, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.500579970 (431h11m14.371663005s) [noise=true , address=45.91.116.12:4228 , piece_id=PVTAKKVVQ427X5LMVEEJQVOO22NKBC6YRALCX3FYXOFJIBZVS3HQ, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.500606741 (431h11m14.371689786s) [seg_pos={0 0}, upload=378516]: Segment reached optimal threshold of 13 pieces. Dec 23 14:40:14.500868379 (431h11m14.371951424s) [noise=true , address=45.91.116.5:4298 , piece_id=LNIEPYC6VOANJJAHWGIR54C2NSQNVBLW56WLJS3ATE3RMQI2WTQQ, seg_pos={0 0}, upload=378516]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: write tcp 188.166.225.170:60010->45.91.116.5:4298: use of closed network connection; write tcp 188.166.225.170:60010->45.91.116.5:4298: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60010->45.91.116.5:4298: use of closed network connection Dec 23 14:40:14.500879801 (431h11m14.371962867s) [noise=true , address=45.91.116.5:4298 , piece_id=LNIEPYC6VOANJJAHWGIR54C2NSQNVBLW56WLJS3ATE3RMQI2WTQQ, seg_pos={0 0}, upload=378516]: Long tail cancellation due to optimal threshold Dec 23 14:40:14.501132863 (431h11m14.372215898s) [noise=true , address=45.91.116.13:4155 , piece_id=BJIO3BI6TOOJLAO7UWOWDEWS2YA7PHTD6XAQZBJ4R344R5FHXIQA, seg_pos={0 0}, upload=378516]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Dec 23 14:40:14.501169122 (431h11m14.372252167s) [noise=true , address=45.91.116.13:4155 , piece_id=BJIO3BI6TOOJLAO7UWOWDEWS2YA7PHTD6XAQZBJ4R344R5FHXIQA, seg_pos={0 0}, upload=378516]: Long tail cancellation due to optimal threshold Dec 23 14:40:14.501199750 (431h11m14.372282786s) [seg_pos={0 0}, upload=378516]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 14:40:14.501347652 (431h11m14.372430707s) [upload=378516]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 14:40:14.501356418 (431h11m14.372439463s) [upload=378516]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 14:40:14.797729085 (431h11m14.66881212s) [upload=378516]: Done uploading segments... Dec 23 14:40:14.797748543 (431h11m14.668831588s) [upload=378516]: Upload finished. err: <nil> Duration: 6.008933606s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.....X............X.........................................] duration: 3.689242448s trace: https://jaeger.dev.storj.io/trace/772f273c480c0b2b FAIL (2025-12-23 14:45:56 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.689269508s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [...X..........X.XX..X............XX..X.............X........] duration: 5.745463901s trace: https://jaeger.dev.storj.io/trace/74788274d19d7a46 FAIL (2025-12-23 14:45:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA, Piece ID: PVTAKKVVQ427X5LMVEEJQVOO22NKBC6YRALCX3FYXOFJIBZVS3HQ) context deadline exceeded; piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: 2QPC7ELY3YBFYD7VUV7RHIPFY5PICWG5ZQMLSZRZKSD43Y6LBBOA) context deadline exceeded ======LOGS====== Dec 23 14:40:11.868119876 (431h11m11.739202911s) [upload=378516]: Starting upload Dec 23 14:40:11.868207653 (431h11m11.739290718s) [upload=378516]: Done starting upload Dec 23 14:40:11.868374521 (431h11m11.739457566s) [upload=378516]: Uploading segments... Dec 23 14:40:11.868506252 (431h11m11.739589297s) [upload=378516]: Got next segment. Inline: false Dec 23 14:40:11.868563651 (431h11m11.739646696s) [upload=378516]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 14:40:11.868569853 (431h11m11.739652898s) [upload=378516]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 14:40:12.111082779 (431h11m11.982165825s) [seg_pos={0 0}, upload=378516]: Begin upload segment... Dec 23 14:40:12.111134018 (431h11m11.982217053s) [seg_pos={0 0}, upload=378516]: Done begin upload segment. Dec 23 14:40:12.111151831 (431h11m11.982234866s) [noise=true , address=145.79.227.41:4085 , piece_id=PJ5H7E2LNKRAEAOFEWIVEV4USLM7DJ4XYD4GSQT5N5SQPWKLRLXA, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.111205434 (431h11m11.982288479s) [noise=true , address=77.90.31.15:4697 , piece_id=LRYC2NIG3MUVOMATXPEG4QMMIWKVED3HCBOQ6C7CXAJCSWVXQVTA, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.111360929 (431h11m11.982443974s) [noise=true , address=145.79.227.39:4013 , piece_id=ENCWF72DYPR54VZN4KXDT3U2CS54BJM2BQQ7BYA5ZP373FDWXZVA, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.111802781 (431h11m11.982885826s) [noise=true , address=77.90.31.13:4627 , piece_id=2QPC7ELY3YBFYD7VUV7RHIPFY5PICWG5ZQMLSZRZKSD43Y6LBBOA, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.112248239 (431h11m11.983331294s) [noise=true , address=45.91.116.13:4155 , piece_id=BJIO3BI6TOOJLAO7UWOWDEWS2YA7PHTD6XAQZBJ4R344R5FHXIQA, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.112648541 (431h11m11.983731586s) [noise=true , address=145.79.227.42:4123 , piece_id=P53WTGLAZA2SBX7MEIMYJABAI76T552NUANBWBNXCDAUNR426CSQ, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.113366068 (431h11m11.984449113s) [noise=true , address=45.91.116.5:4298 , piece_id=LNIEPYC6VOANJJAHWGIR54C2NSQNVBLW56WLJS3ATE3RMQI2WTQQ, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.113765128 (431h11m11.984848163s) [noise=true , address=45.91.116.9:4261 , piece_id=PZIQACPNQHHQD4MILGT6KPOOKU4D6O5FPG4D4TWQOVYLCGOPPAGQ, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.114301239 (431h11m11.985384304s) [noise=true , address=145.79.227.37:4565 , piece_id=K62BRPWTKPSD4IJD2NS2P7QC4JZ3QNLH42GO7XZKI2EAXF62K4KQ, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.114637028 (431h11m11.985720063s) [noise=true , address=45.91.116.12:4228 , piece_id=PVTAKKVVQ427X5LMVEEJQVOO22NKBC6YRALCX3FYXOFJIBZVS3HQ, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.115428274 (431h11m11.986511319s) [noise=true , address=145.79.227.49:4339 , piece_id=PQFUWEUSC7MOP6NNXIOE5M4DFA5OIVRLI3NW2DA6DATVCWQMUGJA, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.115717145 (431h11m11.98680018s) [noise=true , address=145.79.227.4:4187 , piece_id=LIXR5F3EXJTP6ALXC3A3V54IKIAK634JCBRONOCFN6RYI5KGOBHQ, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.116286669 (431h11m11.987369714s) [noise=true , address=145.79.227.50:4373 , piece_id=63RBNMEQ2IDHDV73J4FXZHB37KECBX6SWVNRLM36V5EPWKYTKYMQ, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.116830806 (431h11m11.987913841s) [noise=true , address=145.79.227.14:4839 , piece_id=KKNTSJISRHZKMTMDT75F2QDQF4EV6PTQ46QQTOLWFOGX5E5LDWOQ, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.117007933 (431h11m11.988090968s) [noise=true , address=145.79.227.45:4231 , piece_id=ZLYVEFQLJXGZ4PJ6HRXJZLPQWDWTOH7YAZTH3CCMVXUANT2DVGMA, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.117627111 (431h11m11.988710146s) [noise=true , address=145.79.227.11:4730 , piece_id=M5ZYH4B3UNIY65JJFGGRVXBR2VIOEVEJSK4AFO6SMBEFX2ZXYXYQ, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:13.362711746 (431h11m13.233794791s) [noise=true , address=77.90.31.15:4697 , piece_id=LRYC2NIG3MUVOMATXPEG4QMMIWKVED3HCBOQ6C7CXAJCSWVXQVTA, seg_pos={0 0}, upload=378516]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: storage node overloaded, request limit: 2000 Dec 23 14:40:13.824993954 (431h11m13.696076989s) [upload=378516]: Next returned nil segment Dec 23 14:40:13.825024922 (431h11m13.696107957s) [upload=378516]: Waiting for error group managing segments... Dec 23 14:40:14.102416555 (431h11m13.9734996s) [noise=true , address=145.79.227.41:4085 , piece_id=PJ5H7E2LNKRAEAOFEWIVEV4USLM7DJ4XYD4GSQT5N5SQPWKLRLXA, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.104229709 (431h11m13.975312744s) [noise=true , address=145.79.227.4:4187 , piece_id=LIXR5F3EXJTP6ALXC3A3V54IKIAK634JCBRONOCFN6RYI5KGOBHQ, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.108343191 (431h11m13.979426236s) [noise=true , address=145.79.227.50:4373 , piece_id=63RBNMEQ2IDHDV73J4FXZHB37KECBX6SWVNRLM36V5EPWKYTKYMQ, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.108974293 (431h11m13.980057338s) [noise=true , address=145.79.227.11:4730 , piece_id=M5ZYH4B3UNIY65JJFGGRVXBR2VIOEVEJSK4AFO6SMBEFX2ZXYXYQ, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.115937562 (431h11m13.987020607s) [noise=true , address=145.79.227.37:4565 , piece_id=K62BRPWTKPSD4IJD2NS2P7QC4JZ3QNLH42GO7XZKI2EAXF62K4KQ, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.124740384 (431h11m13.995823419s) [noise=true , address=145.79.227.39:4013 , piece_id=ENCWF72DYPR54VZN4KXDT3U2CS54BJM2BQQ7BYA5ZP373FDWXZVA, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.127790853 (431h11m13.998873898s) [noise=true , address=145.79.227.42:4123 , piece_id=P53WTGLAZA2SBX7MEIMYJABAI76T552NUANBWBNXCDAUNR426CSQ, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.185052439 (431h11m14.056135484s) [noise=true , address=145.79.227.49:4339 , piece_id=PQFUWEUSC7MOP6NNXIOE5M4DFA5OIVRLI3NW2DA6DATVCWQMUGJA, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.190499301 (431h11m14.061582346s) [noise=true , address=145.79.227.45:4231 , piece_id=ZLYVEFQLJXGZ4PJ6HRXJZLPQWDWTOH7YAZTH3CCMVXUANT2DVGMA, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.190648225 (431h11m14.06173127s) [noise=true , address=145.79.227.14:4839 , piece_id=KKNTSJISRHZKMTMDT75F2QDQF4EV6PTQ46QQTOLWFOGX5E5LDWOQ, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.382679737 (431h11m14.253762772s) [noise=true , address=45.91.116.9:4261 , piece_id=PZIQACPNQHHQD4MILGT6KPOOKU4D6O5FPG4D4TWQOVYLCGOPPAGQ, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.487075846 (431h11m14.358158881s) [noise=true , address=77.90.31.13:4627 , piece_id=2QPC7ELY3YBFYD7VUV7RHIPFY5PICWG5ZQMLSZRZKSD43Y6LBBOA, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.500579970 (431h11m14.371663005s) [noise=true , address=45.91.116.12:4228 , piece_id=PVTAKKVVQ427X5LMVEEJQVOO22NKBC6YRALCX3FYXOFJIBZVS3HQ, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.500606741 (431h11m14.371689786s) [seg_pos={0 0}, upload=378516]: Segment reached optimal threshold of 13 pieces. Dec 23 14:40:14.500868379 (431h11m14.371951424s) [noise=true , address=45.91.116.5:4298 , piece_id=LNIEPYC6VOANJJAHWGIR54C2NSQNVBLW56WLJS3ATE3RMQI2WTQQ, seg_pos={0 0}, upload=378516]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: write tcp 188.166.225.170:60010->45.91.116.5:4298: use of closed network connection; write tcp 188.166.225.170:60010->45.91.116.5:4298: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60010->45.91.116.5:4298: use of closed network connection Dec 23 14:40:14.500879801 (431h11m14.371962867s) [noise=true , address=45.91.116.5:4298 , piece_id=LNIEPYC6VOANJJAHWGIR54C2NSQNVBLW56WLJS3ATE3RMQI2WTQQ, seg_pos={0 0}, upload=378516]: Long tail cancellation due to optimal threshold Dec 23 14:40:14.501132863 (431h11m14.372215898s) [noise=true , address=45.91.116.13:4155 , piece_id=BJIO3BI6TOOJLAO7UWOWDEWS2YA7PHTD6XAQZBJ4R344R5FHXIQA, seg_pos={0 0}, upload=378516]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Dec 23 14:40:14.501169122 (431h11m14.372252167s) [noise=true , address=45.91.116.13:4155 , piece_id=BJIO3BI6TOOJLAO7UWOWDEWS2YA7PHTD6XAQZBJ4R344R5FHXIQA, seg_pos={0 0}, upload=378516]: Long tail cancellation due to optimal threshold Dec 23 14:40:14.501199750 (431h11m14.372282786s) [seg_pos={0 0}, upload=378516]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 14:40:14.501347652 (431h11m14.372430707s) [upload=378516]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 14:40:14.501356418 (431h11m14.372439463s) [upload=378516]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 14:40:14.797729085 (431h11m14.66881212s) [upload=378516]: Done uploading segments... Dec 23 14:40:14.797748543 (431h11m14.668831588s) [upload=378516]: Upload finished. err: <nil> Duration: 5.745496124s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [X.....X............X........................................] duration: 3.569476673s trace: https://jaeger.dev.storj.io/trace/772f273c480c0b2b FAIL (2025-12-23 14:45:19 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.569504156s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [....X..........X.XX..X............XX..X.............X.......] duration: 5.861648439s trace: https://jaeger.dev.storj.io/trace/74788274d19d7a46 FAIL (2025-12-23 14:45:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA, Piece ID: PVTAKKVVQ427X5LMVEEJQVOO22NKBC6YRALCX3FYXOFJIBZVS3HQ) context deadline exceeded; piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: 2QPC7ELY3YBFYD7VUV7RHIPFY5PICWG5ZQMLSZRZKSD43Y6LBBOA) context deadline exceeded ======LOGS====== Dec 23 14:40:11.868119876 (431h11m11.739202911s) [upload=378516]: Starting upload Dec 23 14:40:11.868207653 (431h11m11.739290718s) [upload=378516]: Done starting upload Dec 23 14:40:11.868374521 (431h11m11.739457566s) [upload=378516]: Uploading segments... Dec 23 14:40:11.868506252 (431h11m11.739589297s) [upload=378516]: Got next segment. Inline: false Dec 23 14:40:11.868563651 (431h11m11.739646696s) [upload=378516]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 14:40:11.868569853 (431h11m11.739652898s) [upload=378516]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 14:40:12.111082779 (431h11m11.982165825s) [seg_pos={0 0}, upload=378516]: Begin upload segment... Dec 23 14:40:12.111134018 (431h11m11.982217053s) [seg_pos={0 0}, upload=378516]: Done begin upload segment. Dec 23 14:40:12.111151831 (431h11m11.982234866s) [noise=true , address=145.79.227.41:4085 , piece_id=PJ5H7E2LNKRAEAOFEWIVEV4USLM7DJ4XYD4GSQT5N5SQPWKLRLXA, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.111205434 (431h11m11.982288479s) [noise=true , address=77.90.31.15:4697 , piece_id=LRYC2NIG3MUVOMATXPEG4QMMIWKVED3HCBOQ6C7CXAJCSWVXQVTA, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.111360929 (431h11m11.982443974s) [noise=true , address=145.79.227.39:4013 , piece_id=ENCWF72DYPR54VZN4KXDT3U2CS54BJM2BQQ7BYA5ZP373FDWXZVA, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.111802781 (431h11m11.982885826s) [noise=true , address=77.90.31.13:4627 , piece_id=2QPC7ELY3YBFYD7VUV7RHIPFY5PICWG5ZQMLSZRZKSD43Y6LBBOA, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.112248239 (431h11m11.983331294s) [noise=true , address=45.91.116.13:4155 , piece_id=BJIO3BI6TOOJLAO7UWOWDEWS2YA7PHTD6XAQZBJ4R344R5FHXIQA, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.112648541 (431h11m11.983731586s) [noise=true , address=145.79.227.42:4123 , piece_id=P53WTGLAZA2SBX7MEIMYJABAI76T552NUANBWBNXCDAUNR426CSQ, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.113366068 (431h11m11.984449113s) [noise=true , address=45.91.116.5:4298 , piece_id=LNIEPYC6VOANJJAHWGIR54C2NSQNVBLW56WLJS3ATE3RMQI2WTQQ, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.113765128 (431h11m11.984848163s) [noise=true , address=45.91.116.9:4261 , piece_id=PZIQACPNQHHQD4MILGT6KPOOKU4D6O5FPG4D4TWQOVYLCGOPPAGQ, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.114301239 (431h11m11.985384304s) [noise=true , address=145.79.227.37:4565 , piece_id=K62BRPWTKPSD4IJD2NS2P7QC4JZ3QNLH42GO7XZKI2EAXF62K4KQ, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.114637028 (431h11m11.985720063s) [noise=true , address=45.91.116.12:4228 , piece_id=PVTAKKVVQ427X5LMVEEJQVOO22NKBC6YRALCX3FYXOFJIBZVS3HQ, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.115428274 (431h11m11.986511319s) [noise=true , address=145.79.227.49:4339 , piece_id=PQFUWEUSC7MOP6NNXIOE5M4DFA5OIVRLI3NW2DA6DATVCWQMUGJA, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.115717145 (431h11m11.98680018s) [noise=true , address=145.79.227.4:4187 , piece_id=LIXR5F3EXJTP6ALXC3A3V54IKIAK634JCBRONOCFN6RYI5KGOBHQ, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.116286669 (431h11m11.987369714s) [noise=true , address=145.79.227.50:4373 , piece_id=63RBNMEQ2IDHDV73J4FXZHB37KECBX6SWVNRLM36V5EPWKYTKYMQ, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.116830806 (431h11m11.987913841s) [noise=true , address=145.79.227.14:4839 , piece_id=KKNTSJISRHZKMTMDT75F2QDQF4EV6PTQ46QQTOLWFOGX5E5LDWOQ, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.117007933 (431h11m11.988090968s) [noise=true , address=145.79.227.45:4231 , piece_id=ZLYVEFQLJXGZ4PJ6HRXJZLPQWDWTOH7YAZTH3CCMVXUANT2DVGMA, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.117627111 (431h11m11.988710146s) [noise=true , address=145.79.227.11:4730 , piece_id=M5ZYH4B3UNIY65JJFGGRVXBR2VIOEVEJSK4AFO6SMBEFX2ZXYXYQ, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:13.362711746 (431h11m13.233794791s) [noise=true , address=77.90.31.15:4697 , piece_id=LRYC2NIG3MUVOMATXPEG4QMMIWKVED3HCBOQ6C7CXAJCSWVXQVTA, seg_pos={0 0}, upload=378516]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: storage node overloaded, request limit: 2000 Dec 23 14:40:13.824993954 (431h11m13.696076989s) [upload=378516]: Next returned nil segment Dec 23 14:40:13.825024922 (431h11m13.696107957s) [upload=378516]: Waiting for error group managing segments... Dec 23 14:40:14.102416555 (431h11m13.9734996s) [noise=true , address=145.79.227.41:4085 , piece_id=PJ5H7E2LNKRAEAOFEWIVEV4USLM7DJ4XYD4GSQT5N5SQPWKLRLXA, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.104229709 (431h11m13.975312744s) [noise=true , address=145.79.227.4:4187 , piece_id=LIXR5F3EXJTP6ALXC3A3V54IKIAK634JCBRONOCFN6RYI5KGOBHQ, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.108343191 (431h11m13.979426236s) [noise=true , address=145.79.227.50:4373 , piece_id=63RBNMEQ2IDHDV73J4FXZHB37KECBX6SWVNRLM36V5EPWKYTKYMQ, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.108974293 (431h11m13.980057338s) [noise=true , address=145.79.227.11:4730 , piece_id=M5ZYH4B3UNIY65JJFGGRVXBR2VIOEVEJSK4AFO6SMBEFX2ZXYXYQ, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.115937562 (431h11m13.987020607s) [noise=true , address=145.79.227.37:4565 , piece_id=K62BRPWTKPSD4IJD2NS2P7QC4JZ3QNLH42GO7XZKI2EAXF62K4KQ, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.124740384 (431h11m13.995823419s) [noise=true , address=145.79.227.39:4013 , piece_id=ENCWF72DYPR54VZN4KXDT3U2CS54BJM2BQQ7BYA5ZP373FDWXZVA, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.127790853 (431h11m13.998873898s) [noise=true , address=145.79.227.42:4123 , piece_id=P53WTGLAZA2SBX7MEIMYJABAI76T552NUANBWBNXCDAUNR426CSQ, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.185052439 (431h11m14.056135484s) [noise=true , address=145.79.227.49:4339 , piece_id=PQFUWEUSC7MOP6NNXIOE5M4DFA5OIVRLI3NW2DA6DATVCWQMUGJA, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.190499301 (431h11m14.061582346s) [noise=true , address=145.79.227.45:4231 , piece_id=ZLYVEFQLJXGZ4PJ6HRXJZLPQWDWTOH7YAZTH3CCMVXUANT2DVGMA, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.190648225 (431h11m14.06173127s) [noise=true , address=145.79.227.14:4839 , piece_id=KKNTSJISRHZKMTMDT75F2QDQF4EV6PTQ46QQTOLWFOGX5E5LDWOQ, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.382679737 (431h11m14.253762772s) [noise=true , address=45.91.116.9:4261 , piece_id=PZIQACPNQHHQD4MILGT6KPOOKU4D6O5FPG4D4TWQOVYLCGOPPAGQ, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.487075846 (431h11m14.358158881s) [noise=true , address=77.90.31.13:4627 , piece_id=2QPC7ELY3YBFYD7VUV7RHIPFY5PICWG5ZQMLSZRZKSD43Y6LBBOA, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.500579970 (431h11m14.371663005s) [noise=true , address=45.91.116.12:4228 , piece_id=PVTAKKVVQ427X5LMVEEJQVOO22NKBC6YRALCX3FYXOFJIBZVS3HQ, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.500606741 (431h11m14.371689786s) [seg_pos={0 0}, upload=378516]: Segment reached optimal threshold of 13 pieces. Dec 23 14:40:14.500868379 (431h11m14.371951424s) [noise=true , address=45.91.116.5:4298 , piece_id=LNIEPYC6VOANJJAHWGIR54C2NSQNVBLW56WLJS3ATE3RMQI2WTQQ, seg_pos={0 0}, upload=378516]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: write tcp 188.166.225.170:60010->45.91.116.5:4298: use of closed network connection; write tcp 188.166.225.170:60010->45.91.116.5:4298: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60010->45.91.116.5:4298: use of closed network connection Dec 23 14:40:14.500879801 (431h11m14.371962867s) [noise=true , address=45.91.116.5:4298 , piece_id=LNIEPYC6VOANJJAHWGIR54C2NSQNVBLW56WLJS3ATE3RMQI2WTQQ, seg_pos={0 0}, upload=378516]: Long tail cancellation due to optimal threshold Dec 23 14:40:14.501132863 (431h11m14.372215898s) [noise=true , address=45.91.116.13:4155 , piece_id=BJIO3BI6TOOJLAO7UWOWDEWS2YA7PHTD6XAQZBJ4R344R5FHXIQA, seg_pos={0 0}, upload=378516]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Dec 23 14:40:14.501169122 (431h11m14.372252167s) [noise=true , address=45.91.116.13:4155 , piece_id=BJIO3BI6TOOJLAO7UWOWDEWS2YA7PHTD6XAQZBJ4R344R5FHXIQA, seg_pos={0 0}, upload=378516]: Long tail cancellation due to optimal threshold Dec 23 14:40:14.501199750 (431h11m14.372282786s) [seg_pos={0 0}, upload=378516]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 14:40:14.501347652 (431h11m14.372430707s) [upload=378516]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 14:40:14.501356418 (431h11m14.372439463s) [upload=378516]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 14:40:14.797729085 (431h11m14.66881212s) [upload=378516]: Done uploading segments... Dec 23 14:40:14.797748543 (431h11m14.668831588s) [upload=378516]: Upload finished. err: <nil> Duration: 5.861678494s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [.....X..........X.XX..X............XX..X.............X......] duration: 6.835960381s trace: https://jaeger.dev.storj.io/trace/74788274d19d7a46 FAIL (2025-12-23 14:44:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA, Piece ID: PVTAKKVVQ427X5LMVEEJQVOO22NKBC6YRALCX3FYXOFJIBZVS3HQ) context deadline exceeded; piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: 2QPC7ELY3YBFYD7VUV7RHIPFY5PICWG5ZQMLSZRZKSD43Y6LBBOA) context deadline exceeded ======LOGS====== Dec 23 14:40:11.868119876 (431h11m11.739202911s) [upload=378516]: Starting upload Dec 23 14:40:11.868207653 (431h11m11.739290718s) [upload=378516]: Done starting upload Dec 23 14:40:11.868374521 (431h11m11.739457566s) [upload=378516]: Uploading segments... Dec 23 14:40:11.868506252 (431h11m11.739589297s) [upload=378516]: Got next segment. Inline: false Dec 23 14:40:11.868563651 (431h11m11.739646696s) [upload=378516]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 14:40:11.868569853 (431h11m11.739652898s) [upload=378516]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 14:40:12.111082779 (431h11m11.982165825s) [seg_pos={0 0}, upload=378516]: Begin upload segment... Dec 23 14:40:12.111134018 (431h11m11.982217053s) [seg_pos={0 0}, upload=378516]: Done begin upload segment. Dec 23 14:40:12.111151831 (431h11m11.982234866s) [noise=true , address=145.79.227.41:4085 , piece_id=PJ5H7E2LNKRAEAOFEWIVEV4USLM7DJ4XYD4GSQT5N5SQPWKLRLXA, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.111205434 (431h11m11.982288479s) [noise=true , address=77.90.31.15:4697 , piece_id=LRYC2NIG3MUVOMATXPEG4QMMIWKVED3HCBOQ6C7CXAJCSWVXQVTA, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.111360929 (431h11m11.982443974s) [noise=true , address=145.79.227.39:4013 , piece_id=ENCWF72DYPR54VZN4KXDT3U2CS54BJM2BQQ7BYA5ZP373FDWXZVA, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.111802781 (431h11m11.982885826s) [noise=true , address=77.90.31.13:4627 , piece_id=2QPC7ELY3YBFYD7VUV7RHIPFY5PICWG5ZQMLSZRZKSD43Y6LBBOA, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.112248239 (431h11m11.983331294s) [noise=true , address=45.91.116.13:4155 , piece_id=BJIO3BI6TOOJLAO7UWOWDEWS2YA7PHTD6XAQZBJ4R344R5FHXIQA, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.112648541 (431h11m11.983731586s) [noise=true , address=145.79.227.42:4123 , piece_id=P53WTGLAZA2SBX7MEIMYJABAI76T552NUANBWBNXCDAUNR426CSQ, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.113366068 (431h11m11.984449113s) [noise=true , address=45.91.116.5:4298 , piece_id=LNIEPYC6VOANJJAHWGIR54C2NSQNVBLW56WLJS3ATE3RMQI2WTQQ, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.113765128 (431h11m11.984848163s) [noise=true , address=45.91.116.9:4261 , piece_id=PZIQACPNQHHQD4MILGT6KPOOKU4D6O5FPG4D4TWQOVYLCGOPPAGQ, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.114301239 (431h11m11.985384304s) [noise=true , address=145.79.227.37:4565 , piece_id=K62BRPWTKPSD4IJD2NS2P7QC4JZ3QNLH42GO7XZKI2EAXF62K4KQ, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.114637028 (431h11m11.985720063s) [noise=true , address=45.91.116.12:4228 , piece_id=PVTAKKVVQ427X5LMVEEJQVOO22NKBC6YRALCX3FYXOFJIBZVS3HQ, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.115428274 (431h11m11.986511319s) [noise=true , address=145.79.227.49:4339 , piece_id=PQFUWEUSC7MOP6NNXIOE5M4DFA5OIVRLI3NW2DA6DATVCWQMUGJA, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.115717145 (431h11m11.98680018s) [noise=true , address=145.79.227.4:4187 , piece_id=LIXR5F3EXJTP6ALXC3A3V54IKIAK634JCBRONOCFN6RYI5KGOBHQ, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.116286669 (431h11m11.987369714s) [noise=true , address=145.79.227.50:4373 , piece_id=63RBNMEQ2IDHDV73J4FXZHB37KECBX6SWVNRLM36V5EPWKYTKYMQ, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.116830806 (431h11m11.987913841s) [noise=true , address=145.79.227.14:4839 , piece_id=KKNTSJISRHZKMTMDT75F2QDQF4EV6PTQ46QQTOLWFOGX5E5LDWOQ, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.117007933 (431h11m11.988090968s) [noise=true , address=145.79.227.45:4231 , piece_id=ZLYVEFQLJXGZ4PJ6HRXJZLPQWDWTOH7YAZTH3CCMVXUANT2DVGMA, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.117627111 (431h11m11.988710146s) [noise=true , address=145.79.227.11:4730 , piece_id=M5ZYH4B3UNIY65JJFGGRVXBR2VIOEVEJSK4AFO6SMBEFX2ZXYXYQ, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:13.362711746 (431h11m13.233794791s) [noise=true , address=77.90.31.15:4697 , piece_id=LRYC2NIG3MUVOMATXPEG4QMMIWKVED3HCBOQ6C7CXAJCSWVXQVTA, seg_pos={0 0}, upload=378516]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: storage node overloaded, request limit: 2000 Dec 23 14:40:13.824993954 (431h11m13.696076989s) [upload=378516]: Next returned nil segment Dec 23 14:40:13.825024922 (431h11m13.696107957s) [upload=378516]: Waiting for error group managing segments... Dec 23 14:40:14.102416555 (431h11m13.9734996s) [noise=true , address=145.79.227.41:4085 , piece_id=PJ5H7E2LNKRAEAOFEWIVEV4USLM7DJ4XYD4GSQT5N5SQPWKLRLXA, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.104229709 (431h11m13.975312744s) [noise=true , address=145.79.227.4:4187 , piece_id=LIXR5F3EXJTP6ALXC3A3V54IKIAK634JCBRONOCFN6RYI5KGOBHQ, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.108343191 (431h11m13.979426236s) [noise=true , address=145.79.227.50:4373 , piece_id=63RBNMEQ2IDHDV73J4FXZHB37KECBX6SWVNRLM36V5EPWKYTKYMQ, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.108974293 (431h11m13.980057338s) [noise=true , address=145.79.227.11:4730 , piece_id=M5ZYH4B3UNIY65JJFGGRVXBR2VIOEVEJSK4AFO6SMBEFX2ZXYXYQ, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.115937562 (431h11m13.987020607s) [noise=true , address=145.79.227.37:4565 , piece_id=K62BRPWTKPSD4IJD2NS2P7QC4JZ3QNLH42GO7XZKI2EAXF62K4KQ, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.124740384 (431h11m13.995823419s) [noise=true , address=145.79.227.39:4013 , piece_id=ENCWF72DYPR54VZN4KXDT3U2CS54BJM2BQQ7BYA5ZP373FDWXZVA, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.127790853 (431h11m13.998873898s) [noise=true , address=145.79.227.42:4123 , piece_id=P53WTGLAZA2SBX7MEIMYJABAI76T552NUANBWBNXCDAUNR426CSQ, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.185052439 (431h11m14.056135484s) [noise=true , address=145.79.227.49:4339 , piece_id=PQFUWEUSC7MOP6NNXIOE5M4DFA5OIVRLI3NW2DA6DATVCWQMUGJA, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.190499301 (431h11m14.061582346s) [noise=true , address=145.79.227.45:4231 , piece_id=ZLYVEFQLJXGZ4PJ6HRXJZLPQWDWTOH7YAZTH3CCMVXUANT2DVGMA, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.190648225 (431h11m14.06173127s) [noise=true , address=145.79.227.14:4839 , piece_id=KKNTSJISRHZKMTMDT75F2QDQF4EV6PTQ46QQTOLWFOGX5E5LDWOQ, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.382679737 (431h11m14.253762772s) [noise=true , address=45.91.116.9:4261 , piece_id=PZIQACPNQHHQD4MILGT6KPOOKU4D6O5FPG4D4TWQOVYLCGOPPAGQ, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.487075846 (431h11m14.358158881s) [noise=true , address=77.90.31.13:4627 , piece_id=2QPC7ELY3YBFYD7VUV7RHIPFY5PICWG5ZQMLSZRZKSD43Y6LBBOA, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.500579970 (431h11m14.371663005s) [noise=true , address=45.91.116.12:4228 , piece_id=PVTAKKVVQ427X5LMVEEJQVOO22NKBC6YRALCX3FYXOFJIBZVS3HQ, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.500606741 (431h11m14.371689786s) [seg_pos={0 0}, upload=378516]: Segment reached optimal threshold of 13 pieces. Dec 23 14:40:14.500868379 (431h11m14.371951424s) [noise=true , address=45.91.116.5:4298 , piece_id=LNIEPYC6VOANJJAHWGIR54C2NSQNVBLW56WLJS3ATE3RMQI2WTQQ, seg_pos={0 0}, upload=378516]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: write tcp 188.166.225.170:60010->45.91.116.5:4298: use of closed network connection; write tcp 188.166.225.170:60010->45.91.116.5:4298: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60010->45.91.116.5:4298: use of closed network connection Dec 23 14:40:14.500879801 (431h11m14.371962867s) [noise=true , address=45.91.116.5:4298 , piece_id=LNIEPYC6VOANJJAHWGIR54C2NSQNVBLW56WLJS3ATE3RMQI2WTQQ, seg_pos={0 0}, upload=378516]: Long tail cancellation due to optimal threshold Dec 23 14:40:14.501132863 (431h11m14.372215898s) [noise=true , address=45.91.116.13:4155 , piece_id=BJIO3BI6TOOJLAO7UWOWDEWS2YA7PHTD6XAQZBJ4R344R5FHXIQA, seg_pos={0 0}, upload=378516]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Dec 23 14:40:14.501169122 (431h11m14.372252167s) [noise=true , address=45.91.116.13:4155 , piece_id=BJIO3BI6TOOJLAO7UWOWDEWS2YA7PHTD6XAQZBJ4R344R5FHXIQA, seg_pos={0 0}, upload=378516]: Long tail cancellation due to optimal threshold Dec 23 14:40:14.501199750 (431h11m14.372282786s) [seg_pos={0 0}, upload=378516]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 14:40:14.501347652 (431h11m14.372430707s) [upload=378516]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 14:40:14.501356418 (431h11m14.372439463s) [upload=378516]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 14:40:14.797729085 (431h11m14.66881212s) [upload=378516]: Done uploading segments... Dec 23 14:40:14.797748543 (431h11m14.668831588s) [upload=378516]: Upload finished. err: <nil> Duration: 6.83599145s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.X.....X............X.......................................] duration: 3.450034033s trace: https://jaeger.dev.storj.io/trace/772f273c480c0b2b FAIL (2025-12-23 14:44:05 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.450062576s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [..X.....X............X......................................] duration: 3.576809645s trace: https://jaeger.dev.storj.io/trace/772f273c480c0b2b FAIL (2025-12-23 14:43:21 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.576834413s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [......X..........X.XX..X............XX..X.............X.....] duration: 6.544237068s trace: https://jaeger.dev.storj.io/trace/74788274d19d7a46 FAIL (2025-12-23 14:43:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA, Piece ID: PVTAKKVVQ427X5LMVEEJQVOO22NKBC6YRALCX3FYXOFJIBZVS3HQ) context deadline exceeded; piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: 2QPC7ELY3YBFYD7VUV7RHIPFY5PICWG5ZQMLSZRZKSD43Y6LBBOA) context deadline exceeded ======LOGS====== Dec 23 14:40:11.868119876 (431h11m11.739202911s) [upload=378516]: Starting upload Dec 23 14:40:11.868207653 (431h11m11.739290718s) [upload=378516]: Done starting upload Dec 23 14:40:11.868374521 (431h11m11.739457566s) [upload=378516]: Uploading segments... Dec 23 14:40:11.868506252 (431h11m11.739589297s) [upload=378516]: Got next segment. Inline: false Dec 23 14:40:11.868563651 (431h11m11.739646696s) [upload=378516]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 14:40:11.868569853 (431h11m11.739652898s) [upload=378516]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 14:40:12.111082779 (431h11m11.982165825s) [seg_pos={0 0}, upload=378516]: Begin upload segment... Dec 23 14:40:12.111134018 (431h11m11.982217053s) [seg_pos={0 0}, upload=378516]: Done begin upload segment. Dec 23 14:40:12.111151831 (431h11m11.982234866s) [noise=true , address=145.79.227.41:4085 , piece_id=PJ5H7E2LNKRAEAOFEWIVEV4USLM7DJ4XYD4GSQT5N5SQPWKLRLXA, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.111205434 (431h11m11.982288479s) [noise=true , address=77.90.31.15:4697 , piece_id=LRYC2NIG3MUVOMATXPEG4QMMIWKVED3HCBOQ6C7CXAJCSWVXQVTA, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.111360929 (431h11m11.982443974s) [noise=true , address=145.79.227.39:4013 , piece_id=ENCWF72DYPR54VZN4KXDT3U2CS54BJM2BQQ7BYA5ZP373FDWXZVA, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.111802781 (431h11m11.982885826s) [noise=true , address=77.90.31.13:4627 , piece_id=2QPC7ELY3YBFYD7VUV7RHIPFY5PICWG5ZQMLSZRZKSD43Y6LBBOA, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.112248239 (431h11m11.983331294s) [noise=true , address=45.91.116.13:4155 , piece_id=BJIO3BI6TOOJLAO7UWOWDEWS2YA7PHTD6XAQZBJ4R344R5FHXIQA, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.112648541 (431h11m11.983731586s) [noise=true , address=145.79.227.42:4123 , piece_id=P53WTGLAZA2SBX7MEIMYJABAI76T552NUANBWBNXCDAUNR426CSQ, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.113366068 (431h11m11.984449113s) [noise=true , address=45.91.116.5:4298 , piece_id=LNIEPYC6VOANJJAHWGIR54C2NSQNVBLW56WLJS3ATE3RMQI2WTQQ, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.113765128 (431h11m11.984848163s) [noise=true , address=45.91.116.9:4261 , piece_id=PZIQACPNQHHQD4MILGT6KPOOKU4D6O5FPG4D4TWQOVYLCGOPPAGQ, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.114301239 (431h11m11.985384304s) [noise=true , address=145.79.227.37:4565 , piece_id=K62BRPWTKPSD4IJD2NS2P7QC4JZ3QNLH42GO7XZKI2EAXF62K4KQ, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.114637028 (431h11m11.985720063s) [noise=true , address=45.91.116.12:4228 , piece_id=PVTAKKVVQ427X5LMVEEJQVOO22NKBC6YRALCX3FYXOFJIBZVS3HQ, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.115428274 (431h11m11.986511319s) [noise=true , address=145.79.227.49:4339 , piece_id=PQFUWEUSC7MOP6NNXIOE5M4DFA5OIVRLI3NW2DA6DATVCWQMUGJA, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.115717145 (431h11m11.98680018s) [noise=true , address=145.79.227.4:4187 , piece_id=LIXR5F3EXJTP6ALXC3A3V54IKIAK634JCBRONOCFN6RYI5KGOBHQ, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.116286669 (431h11m11.987369714s) [noise=true , address=145.79.227.50:4373 , piece_id=63RBNMEQ2IDHDV73J4FXZHB37KECBX6SWVNRLM36V5EPWKYTKYMQ, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.116830806 (431h11m11.987913841s) [noise=true , address=145.79.227.14:4839 , piece_id=KKNTSJISRHZKMTMDT75F2QDQF4EV6PTQ46QQTOLWFOGX5E5LDWOQ, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.117007933 (431h11m11.988090968s) [noise=true , address=145.79.227.45:4231 , piece_id=ZLYVEFQLJXGZ4PJ6HRXJZLPQWDWTOH7YAZTH3CCMVXUANT2DVGMA, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.117627111 (431h11m11.988710146s) [noise=true , address=145.79.227.11:4730 , piece_id=M5ZYH4B3UNIY65JJFGGRVXBR2VIOEVEJSK4AFO6SMBEFX2ZXYXYQ, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:13.362711746 (431h11m13.233794791s) [noise=true , address=77.90.31.15:4697 , piece_id=LRYC2NIG3MUVOMATXPEG4QMMIWKVED3HCBOQ6C7CXAJCSWVXQVTA, seg_pos={0 0}, upload=378516]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: storage node overloaded, request limit: 2000 Dec 23 14:40:13.824993954 (431h11m13.696076989s) [upload=378516]: Next returned nil segment Dec 23 14:40:13.825024922 (431h11m13.696107957s) [upload=378516]: Waiting for error group managing segments... Dec 23 14:40:14.102416555 (431h11m13.9734996s) [noise=true , address=145.79.227.41:4085 , piece_id=PJ5H7E2LNKRAEAOFEWIVEV4USLM7DJ4XYD4GSQT5N5SQPWKLRLXA, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.104229709 (431h11m13.975312744s) [noise=true , address=145.79.227.4:4187 , piece_id=LIXR5F3EXJTP6ALXC3A3V54IKIAK634JCBRONOCFN6RYI5KGOBHQ, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.108343191 (431h11m13.979426236s) [noise=true , address=145.79.227.50:4373 , piece_id=63RBNMEQ2IDHDV73J4FXZHB37KECBX6SWVNRLM36V5EPWKYTKYMQ, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.108974293 (431h11m13.980057338s) [noise=true , address=145.79.227.11:4730 , piece_id=M5ZYH4B3UNIY65JJFGGRVXBR2VIOEVEJSK4AFO6SMBEFX2ZXYXYQ, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.115937562 (431h11m13.987020607s) [noise=true , address=145.79.227.37:4565 , piece_id=K62BRPWTKPSD4IJD2NS2P7QC4JZ3QNLH42GO7XZKI2EAXF62K4KQ, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.124740384 (431h11m13.995823419s) [noise=true , address=145.79.227.39:4013 , piece_id=ENCWF72DYPR54VZN4KXDT3U2CS54BJM2BQQ7BYA5ZP373FDWXZVA, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.127790853 (431h11m13.998873898s) [noise=true , address=145.79.227.42:4123 , piece_id=P53WTGLAZA2SBX7MEIMYJABAI76T552NUANBWBNXCDAUNR426CSQ, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.185052439 (431h11m14.056135484s) [noise=true , address=145.79.227.49:4339 , piece_id=PQFUWEUSC7MOP6NNXIOE5M4DFA5OIVRLI3NW2DA6DATVCWQMUGJA, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.190499301 (431h11m14.061582346s) [noise=true , address=145.79.227.45:4231 , piece_id=ZLYVEFQLJXGZ4PJ6HRXJZLPQWDWTOH7YAZTH3CCMVXUANT2DVGMA, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.190648225 (431h11m14.06173127s) [noise=true , address=145.79.227.14:4839 , piece_id=KKNTSJISRHZKMTMDT75F2QDQF4EV6PTQ46QQTOLWFOGX5E5LDWOQ, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.382679737 (431h11m14.253762772s) [noise=true , address=45.91.116.9:4261 , piece_id=PZIQACPNQHHQD4MILGT6KPOOKU4D6O5FPG4D4TWQOVYLCGOPPAGQ, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.487075846 (431h11m14.358158881s) [noise=true , address=77.90.31.13:4627 , piece_id=2QPC7ELY3YBFYD7VUV7RHIPFY5PICWG5ZQMLSZRZKSD43Y6LBBOA, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.500579970 (431h11m14.371663005s) [noise=true , address=45.91.116.12:4228 , piece_id=PVTAKKVVQ427X5LMVEEJQVOO22NKBC6YRALCX3FYXOFJIBZVS3HQ, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.500606741 (431h11m14.371689786s) [seg_pos={0 0}, upload=378516]: Segment reached optimal threshold of 13 pieces. Dec 23 14:40:14.500868379 (431h11m14.371951424s) [noise=true , address=45.91.116.5:4298 , piece_id=LNIEPYC6VOANJJAHWGIR54C2NSQNVBLW56WLJS3ATE3RMQI2WTQQ, seg_pos={0 0}, upload=378516]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: write tcp 188.166.225.170:60010->45.91.116.5:4298: use of closed network connection; write tcp 188.166.225.170:60010->45.91.116.5:4298: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60010->45.91.116.5:4298: use of closed network connection Dec 23 14:40:14.500879801 (431h11m14.371962867s) [noise=true , address=45.91.116.5:4298 , piece_id=LNIEPYC6VOANJJAHWGIR54C2NSQNVBLW56WLJS3ATE3RMQI2WTQQ, seg_pos={0 0}, upload=378516]: Long tail cancellation due to optimal threshold Dec 23 14:40:14.501132863 (431h11m14.372215898s) [noise=true , address=45.91.116.13:4155 , piece_id=BJIO3BI6TOOJLAO7UWOWDEWS2YA7PHTD6XAQZBJ4R344R5FHXIQA, seg_pos={0 0}, upload=378516]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Dec 23 14:40:14.501169122 (431h11m14.372252167s) [noise=true , address=45.91.116.13:4155 , piece_id=BJIO3BI6TOOJLAO7UWOWDEWS2YA7PHTD6XAQZBJ4R344R5FHXIQA, seg_pos={0 0}, upload=378516]: Long tail cancellation due to optimal threshold Dec 23 14:40:14.501199750 (431h11m14.372282786s) [seg_pos={0 0}, upload=378516]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 14:40:14.501347652 (431h11m14.372430707s) [upload=378516]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 14:40:14.501356418 (431h11m14.372439463s) [upload=378516]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 14:40:14.797729085 (431h11m14.66881212s) [upload=378516]: Done uploading segments... Dec 23 14:40:14.797748543 (431h11m14.668831588s) [upload=378516]: Upload finished. err: <nil> Duration: 6.544257277s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [.......X..........X.XX..X............XX..X.............X....] duration: 6.011799652s trace: https://jaeger.dev.storj.io/trace/74788274d19d7a46 FAIL (2025-12-23 14:42:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA, Piece ID: PVTAKKVVQ427X5LMVEEJQVOO22NKBC6YRALCX3FYXOFJIBZVS3HQ) context deadline exceeded; piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: 2QPC7ELY3YBFYD7VUV7RHIPFY5PICWG5ZQMLSZRZKSD43Y6LBBOA) context deadline exceeded ======LOGS====== Dec 23 14:40:11.868119876 (431h11m11.739202911s) [upload=378516]: Starting upload Dec 23 14:40:11.868207653 (431h11m11.739290718s) [upload=378516]: Done starting upload Dec 23 14:40:11.868374521 (431h11m11.739457566s) [upload=378516]: Uploading segments... Dec 23 14:40:11.868506252 (431h11m11.739589297s) [upload=378516]: Got next segment. Inline: false Dec 23 14:40:11.868563651 (431h11m11.739646696s) [upload=378516]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 14:40:11.868569853 (431h11m11.739652898s) [upload=378516]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 14:40:12.111082779 (431h11m11.982165825s) [seg_pos={0 0}, upload=378516]: Begin upload segment... Dec 23 14:40:12.111134018 (431h11m11.982217053s) [seg_pos={0 0}, upload=378516]: Done begin upload segment. Dec 23 14:40:12.111151831 (431h11m11.982234866s) [noise=true , address=145.79.227.41:4085 , piece_id=PJ5H7E2LNKRAEAOFEWIVEV4USLM7DJ4XYD4GSQT5N5SQPWKLRLXA, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.111205434 (431h11m11.982288479s) [noise=true , address=77.90.31.15:4697 , piece_id=LRYC2NIG3MUVOMATXPEG4QMMIWKVED3HCBOQ6C7CXAJCSWVXQVTA, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.111360929 (431h11m11.982443974s) [noise=true , address=145.79.227.39:4013 , piece_id=ENCWF72DYPR54VZN4KXDT3U2CS54BJM2BQQ7BYA5ZP373FDWXZVA, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.111802781 (431h11m11.982885826s) [noise=true , address=77.90.31.13:4627 , piece_id=2QPC7ELY3YBFYD7VUV7RHIPFY5PICWG5ZQMLSZRZKSD43Y6LBBOA, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.112248239 (431h11m11.983331294s) [noise=true , address=45.91.116.13:4155 , piece_id=BJIO3BI6TOOJLAO7UWOWDEWS2YA7PHTD6XAQZBJ4R344R5FHXIQA, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.112648541 (431h11m11.983731586s) [noise=true , address=145.79.227.42:4123 , piece_id=P53WTGLAZA2SBX7MEIMYJABAI76T552NUANBWBNXCDAUNR426CSQ, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.113366068 (431h11m11.984449113s) [noise=true , address=45.91.116.5:4298 , piece_id=LNIEPYC6VOANJJAHWGIR54C2NSQNVBLW56WLJS3ATE3RMQI2WTQQ, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.113765128 (431h11m11.984848163s) [noise=true , address=45.91.116.9:4261 , piece_id=PZIQACPNQHHQD4MILGT6KPOOKU4D6O5FPG4D4TWQOVYLCGOPPAGQ, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.114301239 (431h11m11.985384304s) [noise=true , address=145.79.227.37:4565 , piece_id=K62BRPWTKPSD4IJD2NS2P7QC4JZ3QNLH42GO7XZKI2EAXF62K4KQ, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.114637028 (431h11m11.985720063s) [noise=true , address=45.91.116.12:4228 , piece_id=PVTAKKVVQ427X5LMVEEJQVOO22NKBC6YRALCX3FYXOFJIBZVS3HQ, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.115428274 (431h11m11.986511319s) [noise=true , address=145.79.227.49:4339 , piece_id=PQFUWEUSC7MOP6NNXIOE5M4DFA5OIVRLI3NW2DA6DATVCWQMUGJA, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.115717145 (431h11m11.98680018s) [noise=true , address=145.79.227.4:4187 , piece_id=LIXR5F3EXJTP6ALXC3A3V54IKIAK634JCBRONOCFN6RYI5KGOBHQ, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.116286669 (431h11m11.987369714s) [noise=true , address=145.79.227.50:4373 , piece_id=63RBNMEQ2IDHDV73J4FXZHB37KECBX6SWVNRLM36V5EPWKYTKYMQ, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.116830806 (431h11m11.987913841s) [noise=true , address=145.79.227.14:4839 , piece_id=KKNTSJISRHZKMTMDT75F2QDQF4EV6PTQ46QQTOLWFOGX5E5LDWOQ, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.117007933 (431h11m11.988090968s) [noise=true , address=145.79.227.45:4231 , piece_id=ZLYVEFQLJXGZ4PJ6HRXJZLPQWDWTOH7YAZTH3CCMVXUANT2DVGMA, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.117627111 (431h11m11.988710146s) [noise=true , address=145.79.227.11:4730 , piece_id=M5ZYH4B3UNIY65JJFGGRVXBR2VIOEVEJSK4AFO6SMBEFX2ZXYXYQ, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:13.362711746 (431h11m13.233794791s) [noise=true , address=77.90.31.15:4697 , piece_id=LRYC2NIG3MUVOMATXPEG4QMMIWKVED3HCBOQ6C7CXAJCSWVXQVTA, seg_pos={0 0}, upload=378516]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: storage node overloaded, request limit: 2000 Dec 23 14:40:13.824993954 (431h11m13.696076989s) [upload=378516]: Next returned nil segment Dec 23 14:40:13.825024922 (431h11m13.696107957s) [upload=378516]: Waiting for error group managing segments... Dec 23 14:40:14.102416555 (431h11m13.9734996s) [noise=true , address=145.79.227.41:4085 , piece_id=PJ5H7E2LNKRAEAOFEWIVEV4USLM7DJ4XYD4GSQT5N5SQPWKLRLXA, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.104229709 (431h11m13.975312744s) [noise=true , address=145.79.227.4:4187 , piece_id=LIXR5F3EXJTP6ALXC3A3V54IKIAK634JCBRONOCFN6RYI5KGOBHQ, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.108343191 (431h11m13.979426236s) [noise=true , address=145.79.227.50:4373 , piece_id=63RBNMEQ2IDHDV73J4FXZHB37KECBX6SWVNRLM36V5EPWKYTKYMQ, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.108974293 (431h11m13.980057338s) [noise=true , address=145.79.227.11:4730 , piece_id=M5ZYH4B3UNIY65JJFGGRVXBR2VIOEVEJSK4AFO6SMBEFX2ZXYXYQ, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.115937562 (431h11m13.987020607s) [noise=true , address=145.79.227.37:4565 , piece_id=K62BRPWTKPSD4IJD2NS2P7QC4JZ3QNLH42GO7XZKI2EAXF62K4KQ, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.124740384 (431h11m13.995823419s) [noise=true , address=145.79.227.39:4013 , piece_id=ENCWF72DYPR54VZN4KXDT3U2CS54BJM2BQQ7BYA5ZP373FDWXZVA, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.127790853 (431h11m13.998873898s) [noise=true , address=145.79.227.42:4123 , piece_id=P53WTGLAZA2SBX7MEIMYJABAI76T552NUANBWBNXCDAUNR426CSQ, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.185052439 (431h11m14.056135484s) [noise=true , address=145.79.227.49:4339 , piece_id=PQFUWEUSC7MOP6NNXIOE5M4DFA5OIVRLI3NW2DA6DATVCWQMUGJA, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.190499301 (431h11m14.061582346s) [noise=true , address=145.79.227.45:4231 , piece_id=ZLYVEFQLJXGZ4PJ6HRXJZLPQWDWTOH7YAZTH3CCMVXUANT2DVGMA, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.190648225 (431h11m14.06173127s) [noise=true , address=145.79.227.14:4839 , piece_id=KKNTSJISRHZKMTMDT75F2QDQF4EV6PTQ46QQTOLWFOGX5E5LDWOQ, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.382679737 (431h11m14.253762772s) [noise=true , address=45.91.116.9:4261 , piece_id=PZIQACPNQHHQD4MILGT6KPOOKU4D6O5FPG4D4TWQOVYLCGOPPAGQ, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.487075846 (431h11m14.358158881s) [noise=true , address=77.90.31.13:4627 , piece_id=2QPC7ELY3YBFYD7VUV7RHIPFY5PICWG5ZQMLSZRZKSD43Y6LBBOA, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.500579970 (431h11m14.371663005s) [noise=true , address=45.91.116.12:4228 , piece_id=PVTAKKVVQ427X5LMVEEJQVOO22NKBC6YRALCX3FYXOFJIBZVS3HQ, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.500606741 (431h11m14.371689786s) [seg_pos={0 0}, upload=378516]: Segment reached optimal threshold of 13 pieces. Dec 23 14:40:14.500868379 (431h11m14.371951424s) [noise=true , address=45.91.116.5:4298 , piece_id=LNIEPYC6VOANJJAHWGIR54C2NSQNVBLW56WLJS3ATE3RMQI2WTQQ, seg_pos={0 0}, upload=378516]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: write tcp 188.166.225.170:60010->45.91.116.5:4298: use of closed network connection; write tcp 188.166.225.170:60010->45.91.116.5:4298: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60010->45.91.116.5:4298: use of closed network connection Dec 23 14:40:14.500879801 (431h11m14.371962867s) [noise=true , address=45.91.116.5:4298 , piece_id=LNIEPYC6VOANJJAHWGIR54C2NSQNVBLW56WLJS3ATE3RMQI2WTQQ, seg_pos={0 0}, upload=378516]: Long tail cancellation due to optimal threshold Dec 23 14:40:14.501132863 (431h11m14.372215898s) [noise=true , address=45.91.116.13:4155 , piece_id=BJIO3BI6TOOJLAO7UWOWDEWS2YA7PHTD6XAQZBJ4R344R5FHXIQA, seg_pos={0 0}, upload=378516]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Dec 23 14:40:14.501169122 (431h11m14.372252167s) [noise=true , address=45.91.116.13:4155 , piece_id=BJIO3BI6TOOJLAO7UWOWDEWS2YA7PHTD6XAQZBJ4R344R5FHXIQA, seg_pos={0 0}, upload=378516]: Long tail cancellation due to optimal threshold Dec 23 14:40:14.501199750 (431h11m14.372282786s) [seg_pos={0 0}, upload=378516]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 14:40:14.501347652 (431h11m14.372430707s) [upload=378516]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 14:40:14.501356418 (431h11m14.372439463s) [upload=378516]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 14:40:14.797729085 (431h11m14.66881212s) [upload=378516]: Done uploading segments... Dec 23 14:40:14.797748543 (431h11m14.668831588s) [upload=378516]: Upload finished. err: <nil> Duration: 6.011952644s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [...X.....X............X.....................................] duration: 3.428881603s trace: https://jaeger.dev.storj.io/trace/772f273c480c0b2b FAIL (2025-12-23 14:42:06 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.428907052s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [........X..........X.XX..X............XX..X.............X...] duration: 5.994196733s trace: https://jaeger.dev.storj.io/trace/74788274d19d7a46 FAIL (2025-12-23 14:41:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA, Piece ID: PVTAKKVVQ427X5LMVEEJQVOO22NKBC6YRALCX3FYXOFJIBZVS3HQ) context deadline exceeded; piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: 2QPC7ELY3YBFYD7VUV7RHIPFY5PICWG5ZQMLSZRZKSD43Y6LBBOA) context deadline exceeded ======LOGS====== Dec 23 14:40:11.868119876 (431h11m11.739202911s) [upload=378516]: Starting upload Dec 23 14:40:11.868207653 (431h11m11.739290718s) [upload=378516]: Done starting upload Dec 23 14:40:11.868374521 (431h11m11.739457566s) [upload=378516]: Uploading segments... Dec 23 14:40:11.868506252 (431h11m11.739589297s) [upload=378516]: Got next segment. Inline: false Dec 23 14:40:11.868563651 (431h11m11.739646696s) [upload=378516]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 14:40:11.868569853 (431h11m11.739652898s) [upload=378516]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 14:40:12.111082779 (431h11m11.982165825s) [seg_pos={0 0}, upload=378516]: Begin upload segment... Dec 23 14:40:12.111134018 (431h11m11.982217053s) [seg_pos={0 0}, upload=378516]: Done begin upload segment. Dec 23 14:40:12.111151831 (431h11m11.982234866s) [noise=true , address=145.79.227.41:4085 , piece_id=PJ5H7E2LNKRAEAOFEWIVEV4USLM7DJ4XYD4GSQT5N5SQPWKLRLXA, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.111205434 (431h11m11.982288479s) [noise=true , address=77.90.31.15:4697 , piece_id=LRYC2NIG3MUVOMATXPEG4QMMIWKVED3HCBOQ6C7CXAJCSWVXQVTA, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.111360929 (431h11m11.982443974s) [noise=true , address=145.79.227.39:4013 , piece_id=ENCWF72DYPR54VZN4KXDT3U2CS54BJM2BQQ7BYA5ZP373FDWXZVA, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.111802781 (431h11m11.982885826s) [noise=true , address=77.90.31.13:4627 , piece_id=2QPC7ELY3YBFYD7VUV7RHIPFY5PICWG5ZQMLSZRZKSD43Y6LBBOA, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.112248239 (431h11m11.983331294s) [noise=true , address=45.91.116.13:4155 , piece_id=BJIO3BI6TOOJLAO7UWOWDEWS2YA7PHTD6XAQZBJ4R344R5FHXIQA, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.112648541 (431h11m11.983731586s) [noise=true , address=145.79.227.42:4123 , piece_id=P53WTGLAZA2SBX7MEIMYJABAI76T552NUANBWBNXCDAUNR426CSQ, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.113366068 (431h11m11.984449113s) [noise=true , address=45.91.116.5:4298 , piece_id=LNIEPYC6VOANJJAHWGIR54C2NSQNVBLW56WLJS3ATE3RMQI2WTQQ, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.113765128 (431h11m11.984848163s) [noise=true , address=45.91.116.9:4261 , piece_id=PZIQACPNQHHQD4MILGT6KPOOKU4D6O5FPG4D4TWQOVYLCGOPPAGQ, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.114301239 (431h11m11.985384304s) [noise=true , address=145.79.227.37:4565 , piece_id=K62BRPWTKPSD4IJD2NS2P7QC4JZ3QNLH42GO7XZKI2EAXF62K4KQ, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.114637028 (431h11m11.985720063s) [noise=true , address=45.91.116.12:4228 , piece_id=PVTAKKVVQ427X5LMVEEJQVOO22NKBC6YRALCX3FYXOFJIBZVS3HQ, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.115428274 (431h11m11.986511319s) [noise=true , address=145.79.227.49:4339 , piece_id=PQFUWEUSC7MOP6NNXIOE5M4DFA5OIVRLI3NW2DA6DATVCWQMUGJA, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.115717145 (431h11m11.98680018s) [noise=true , address=145.79.227.4:4187 , piece_id=LIXR5F3EXJTP6ALXC3A3V54IKIAK634JCBRONOCFN6RYI5KGOBHQ, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.116286669 (431h11m11.987369714s) [noise=true , address=145.79.227.50:4373 , piece_id=63RBNMEQ2IDHDV73J4FXZHB37KECBX6SWVNRLM36V5EPWKYTKYMQ, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.116830806 (431h11m11.987913841s) [noise=true , address=145.79.227.14:4839 , piece_id=KKNTSJISRHZKMTMDT75F2QDQF4EV6PTQ46QQTOLWFOGX5E5LDWOQ, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.117007933 (431h11m11.988090968s) [noise=true , address=145.79.227.45:4231 , piece_id=ZLYVEFQLJXGZ4PJ6HRXJZLPQWDWTOH7YAZTH3CCMVXUANT2DVGMA, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.117627111 (431h11m11.988710146s) [noise=true , address=145.79.227.11:4730 , piece_id=M5ZYH4B3UNIY65JJFGGRVXBR2VIOEVEJSK4AFO6SMBEFX2ZXYXYQ, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:13.362711746 (431h11m13.233794791s) [noise=true , address=77.90.31.15:4697 , piece_id=LRYC2NIG3MUVOMATXPEG4QMMIWKVED3HCBOQ6C7CXAJCSWVXQVTA, seg_pos={0 0}, upload=378516]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: storage node overloaded, request limit: 2000 Dec 23 14:40:13.824993954 (431h11m13.696076989s) [upload=378516]: Next returned nil segment Dec 23 14:40:13.825024922 (431h11m13.696107957s) [upload=378516]: Waiting for error group managing segments... Dec 23 14:40:14.102416555 (431h11m13.9734996s) [noise=true , address=145.79.227.41:4085 , piece_id=PJ5H7E2LNKRAEAOFEWIVEV4USLM7DJ4XYD4GSQT5N5SQPWKLRLXA, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.104229709 (431h11m13.975312744s) [noise=true , address=145.79.227.4:4187 , piece_id=LIXR5F3EXJTP6ALXC3A3V54IKIAK634JCBRONOCFN6RYI5KGOBHQ, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.108343191 (431h11m13.979426236s) [noise=true , address=145.79.227.50:4373 , piece_id=63RBNMEQ2IDHDV73J4FXZHB37KECBX6SWVNRLM36V5EPWKYTKYMQ, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.108974293 (431h11m13.980057338s) [noise=true , address=145.79.227.11:4730 , piece_id=M5ZYH4B3UNIY65JJFGGRVXBR2VIOEVEJSK4AFO6SMBEFX2ZXYXYQ, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.115937562 (431h11m13.987020607s) [noise=true , address=145.79.227.37:4565 , piece_id=K62BRPWTKPSD4IJD2NS2P7QC4JZ3QNLH42GO7XZKI2EAXF62K4KQ, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.124740384 (431h11m13.995823419s) [noise=true , address=145.79.227.39:4013 , piece_id=ENCWF72DYPR54VZN4KXDT3U2CS54BJM2BQQ7BYA5ZP373FDWXZVA, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.127790853 (431h11m13.998873898s) [noise=true , address=145.79.227.42:4123 , piece_id=P53WTGLAZA2SBX7MEIMYJABAI76T552NUANBWBNXCDAUNR426CSQ, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.185052439 (431h11m14.056135484s) [noise=true , address=145.79.227.49:4339 , piece_id=PQFUWEUSC7MOP6NNXIOE5M4DFA5OIVRLI3NW2DA6DATVCWQMUGJA, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.190499301 (431h11m14.061582346s) [noise=true , address=145.79.227.45:4231 , piece_id=ZLYVEFQLJXGZ4PJ6HRXJZLPQWDWTOH7YAZTH3CCMVXUANT2DVGMA, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.190648225 (431h11m14.06173127s) [noise=true , address=145.79.227.14:4839 , piece_id=KKNTSJISRHZKMTMDT75F2QDQF4EV6PTQ46QQTOLWFOGX5E5LDWOQ, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.382679737 (431h11m14.253762772s) [noise=true , address=45.91.116.9:4261 , piece_id=PZIQACPNQHHQD4MILGT6KPOOKU4D6O5FPG4D4TWQOVYLCGOPPAGQ, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.487075846 (431h11m14.358158881s) [noise=true , address=77.90.31.13:4627 , piece_id=2QPC7ELY3YBFYD7VUV7RHIPFY5PICWG5ZQMLSZRZKSD43Y6LBBOA, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.500579970 (431h11m14.371663005s) [noise=true , address=45.91.116.12:4228 , piece_id=PVTAKKVVQ427X5LMVEEJQVOO22NKBC6YRALCX3FYXOFJIBZVS3HQ, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.500606741 (431h11m14.371689786s) [seg_pos={0 0}, upload=378516]: Segment reached optimal threshold of 13 pieces. Dec 23 14:40:14.500868379 (431h11m14.371951424s) [noise=true , address=45.91.116.5:4298 , piece_id=LNIEPYC6VOANJJAHWGIR54C2NSQNVBLW56WLJS3ATE3RMQI2WTQQ, seg_pos={0 0}, upload=378516]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: write tcp 188.166.225.170:60010->45.91.116.5:4298: use of closed network connection; write tcp 188.166.225.170:60010->45.91.116.5:4298: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60010->45.91.116.5:4298: use of closed network connection Dec 23 14:40:14.500879801 (431h11m14.371962867s) [noise=true , address=45.91.116.5:4298 , piece_id=LNIEPYC6VOANJJAHWGIR54C2NSQNVBLW56WLJS3ATE3RMQI2WTQQ, seg_pos={0 0}, upload=378516]: Long tail cancellation due to optimal threshold Dec 23 14:40:14.501132863 (431h11m14.372215898s) [noise=true , address=45.91.116.13:4155 , piece_id=BJIO3BI6TOOJLAO7UWOWDEWS2YA7PHTD6XAQZBJ4R344R5FHXIQA, seg_pos={0 0}, upload=378516]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Dec 23 14:40:14.501169122 (431h11m14.372252167s) [noise=true , address=45.91.116.13:4155 , piece_id=BJIO3BI6TOOJLAO7UWOWDEWS2YA7PHTD6XAQZBJ4R344R5FHXIQA, seg_pos={0 0}, upload=378516]: Long tail cancellation due to optimal threshold Dec 23 14:40:14.501199750 (431h11m14.372282786s) [seg_pos={0 0}, upload=378516]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 14:40:14.501347652 (431h11m14.372430707s) [upload=378516]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 14:40:14.501356418 (431h11m14.372439463s) [upload=378516]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 14:40:14.797729085 (431h11m14.66881212s) [upload=378516]: Done uploading segments... Dec 23 14:40:14.797748543 (431h11m14.668831588s) [upload=378516]: Upload finished. err: <nil> Duration: 5.994223996s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [.........X..........X.XX..X............XX..X.............X..] duration: 6.351808831s trace: https://jaeger.dev.storj.io/trace/74788274d19d7a46 FAIL (2025-12-23 14:41:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA, Piece ID: PVTAKKVVQ427X5LMVEEJQVOO22NKBC6YRALCX3FYXOFJIBZVS3HQ) context deadline exceeded; piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: 2QPC7ELY3YBFYD7VUV7RHIPFY5PICWG5ZQMLSZRZKSD43Y6LBBOA) context deadline exceeded ======LOGS====== Dec 23 14:40:11.868119876 (431h11m11.739202911s) [upload=378516]: Starting upload Dec 23 14:40:11.868207653 (431h11m11.739290718s) [upload=378516]: Done starting upload Dec 23 14:40:11.868374521 (431h11m11.739457566s) [upload=378516]: Uploading segments... Dec 23 14:40:11.868506252 (431h11m11.739589297s) [upload=378516]: Got next segment. Inline: false Dec 23 14:40:11.868563651 (431h11m11.739646696s) [upload=378516]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 14:40:11.868569853 (431h11m11.739652898s) [upload=378516]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 14:40:12.111082779 (431h11m11.982165825s) [seg_pos={0 0}, upload=378516]: Begin upload segment... Dec 23 14:40:12.111134018 (431h11m11.982217053s) [seg_pos={0 0}, upload=378516]: Done begin upload segment. Dec 23 14:40:12.111151831 (431h11m11.982234866s) [noise=true , address=145.79.227.41:4085 , piece_id=PJ5H7E2LNKRAEAOFEWIVEV4USLM7DJ4XYD4GSQT5N5SQPWKLRLXA, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.111205434 (431h11m11.982288479s) [noise=true , address=77.90.31.15:4697 , piece_id=LRYC2NIG3MUVOMATXPEG4QMMIWKVED3HCBOQ6C7CXAJCSWVXQVTA, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.111360929 (431h11m11.982443974s) [noise=true , address=145.79.227.39:4013 , piece_id=ENCWF72DYPR54VZN4KXDT3U2CS54BJM2BQQ7BYA5ZP373FDWXZVA, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.111802781 (431h11m11.982885826s) [noise=true , address=77.90.31.13:4627 , piece_id=2QPC7ELY3YBFYD7VUV7RHIPFY5PICWG5ZQMLSZRZKSD43Y6LBBOA, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.112248239 (431h11m11.983331294s) [noise=true , address=45.91.116.13:4155 , piece_id=BJIO3BI6TOOJLAO7UWOWDEWS2YA7PHTD6XAQZBJ4R344R5FHXIQA, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.112648541 (431h11m11.983731586s) [noise=true , address=145.79.227.42:4123 , piece_id=P53WTGLAZA2SBX7MEIMYJABAI76T552NUANBWBNXCDAUNR426CSQ, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.113366068 (431h11m11.984449113s) [noise=true , address=45.91.116.5:4298 , piece_id=LNIEPYC6VOANJJAHWGIR54C2NSQNVBLW56WLJS3ATE3RMQI2WTQQ, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.113765128 (431h11m11.984848163s) [noise=true , address=45.91.116.9:4261 , piece_id=PZIQACPNQHHQD4MILGT6KPOOKU4D6O5FPG4D4TWQOVYLCGOPPAGQ, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.114301239 (431h11m11.985384304s) [noise=true , address=145.79.227.37:4565 , piece_id=K62BRPWTKPSD4IJD2NS2P7QC4JZ3QNLH42GO7XZKI2EAXF62K4KQ, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.114637028 (431h11m11.985720063s) [noise=true , address=45.91.116.12:4228 , piece_id=PVTAKKVVQ427X5LMVEEJQVOO22NKBC6YRALCX3FYXOFJIBZVS3HQ, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.115428274 (431h11m11.986511319s) [noise=true , address=145.79.227.49:4339 , piece_id=PQFUWEUSC7MOP6NNXIOE5M4DFA5OIVRLI3NW2DA6DATVCWQMUGJA, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.115717145 (431h11m11.98680018s) [noise=true , address=145.79.227.4:4187 , piece_id=LIXR5F3EXJTP6ALXC3A3V54IKIAK634JCBRONOCFN6RYI5KGOBHQ, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.116286669 (431h11m11.987369714s) [noise=true , address=145.79.227.50:4373 , piece_id=63RBNMEQ2IDHDV73J4FXZHB37KECBX6SWVNRLM36V5EPWKYTKYMQ, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.116830806 (431h11m11.987913841s) [noise=true , address=145.79.227.14:4839 , piece_id=KKNTSJISRHZKMTMDT75F2QDQF4EV6PTQ46QQTOLWFOGX5E5LDWOQ, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.117007933 (431h11m11.988090968s) [noise=true , address=145.79.227.45:4231 , piece_id=ZLYVEFQLJXGZ4PJ6HRXJZLPQWDWTOH7YAZTH3CCMVXUANT2DVGMA, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.117627111 (431h11m11.988710146s) [noise=true , address=145.79.227.11:4730 , piece_id=M5ZYH4B3UNIY65JJFGGRVXBR2VIOEVEJSK4AFO6SMBEFX2ZXYXYQ, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:13.362711746 (431h11m13.233794791s) [noise=true , address=77.90.31.15:4697 , piece_id=LRYC2NIG3MUVOMATXPEG4QMMIWKVED3HCBOQ6C7CXAJCSWVXQVTA, seg_pos={0 0}, upload=378516]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: storage node overloaded, request limit: 2000 Dec 23 14:40:13.824993954 (431h11m13.696076989s) [upload=378516]: Next returned nil segment Dec 23 14:40:13.825024922 (431h11m13.696107957s) [upload=378516]: Waiting for error group managing segments... Dec 23 14:40:14.102416555 (431h11m13.9734996s) [noise=true , address=145.79.227.41:4085 , piece_id=PJ5H7E2LNKRAEAOFEWIVEV4USLM7DJ4XYD4GSQT5N5SQPWKLRLXA, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.104229709 (431h11m13.975312744s) [noise=true , address=145.79.227.4:4187 , piece_id=LIXR5F3EXJTP6ALXC3A3V54IKIAK634JCBRONOCFN6RYI5KGOBHQ, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.108343191 (431h11m13.979426236s) [noise=true , address=145.79.227.50:4373 , piece_id=63RBNMEQ2IDHDV73J4FXZHB37KECBX6SWVNRLM36V5EPWKYTKYMQ, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.108974293 (431h11m13.980057338s) [noise=true , address=145.79.227.11:4730 , piece_id=M5ZYH4B3UNIY65JJFGGRVXBR2VIOEVEJSK4AFO6SMBEFX2ZXYXYQ, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.115937562 (431h11m13.987020607s) [noise=true , address=145.79.227.37:4565 , piece_id=K62BRPWTKPSD4IJD2NS2P7QC4JZ3QNLH42GO7XZKI2EAXF62K4KQ, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.124740384 (431h11m13.995823419s) [noise=true , address=145.79.227.39:4013 , piece_id=ENCWF72DYPR54VZN4KXDT3U2CS54BJM2BQQ7BYA5ZP373FDWXZVA, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.127790853 (431h11m13.998873898s) [noise=true , address=145.79.227.42:4123 , piece_id=P53WTGLAZA2SBX7MEIMYJABAI76T552NUANBWBNXCDAUNR426CSQ, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.185052439 (431h11m14.056135484s) [noise=true , address=145.79.227.49:4339 , piece_id=PQFUWEUSC7MOP6NNXIOE5M4DFA5OIVRLI3NW2DA6DATVCWQMUGJA, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.190499301 (431h11m14.061582346s) [noise=true , address=145.79.227.45:4231 , piece_id=ZLYVEFQLJXGZ4PJ6HRXJZLPQWDWTOH7YAZTH3CCMVXUANT2DVGMA, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.190648225 (431h11m14.06173127s) [noise=true , address=145.79.227.14:4839 , piece_id=KKNTSJISRHZKMTMDT75F2QDQF4EV6PTQ46QQTOLWFOGX5E5LDWOQ, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.382679737 (431h11m14.253762772s) [noise=true , address=45.91.116.9:4261 , piece_id=PZIQACPNQHHQD4MILGT6KPOOKU4D6O5FPG4D4TWQOVYLCGOPPAGQ, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.487075846 (431h11m14.358158881s) [noise=true , address=77.90.31.13:4627 , piece_id=2QPC7ELY3YBFYD7VUV7RHIPFY5PICWG5ZQMLSZRZKSD43Y6LBBOA, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.500579970 (431h11m14.371663005s) [noise=true , address=45.91.116.12:4228 , piece_id=PVTAKKVVQ427X5LMVEEJQVOO22NKBC6YRALCX3FYXOFJIBZVS3HQ, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.500606741 (431h11m14.371689786s) [seg_pos={0 0}, upload=378516]: Segment reached optimal threshold of 13 pieces. Dec 23 14:40:14.500868379 (431h11m14.371951424s) [noise=true , address=45.91.116.5:4298 , piece_id=LNIEPYC6VOANJJAHWGIR54C2NSQNVBLW56WLJS3ATE3RMQI2WTQQ, seg_pos={0 0}, upload=378516]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: write tcp 188.166.225.170:60010->45.91.116.5:4298: use of closed network connection; write tcp 188.166.225.170:60010->45.91.116.5:4298: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60010->45.91.116.5:4298: use of closed network connection Dec 23 14:40:14.500879801 (431h11m14.371962867s) [noise=true , address=45.91.116.5:4298 , piece_id=LNIEPYC6VOANJJAHWGIR54C2NSQNVBLW56WLJS3ATE3RMQI2WTQQ, seg_pos={0 0}, upload=378516]: Long tail cancellation due to optimal threshold Dec 23 14:40:14.501132863 (431h11m14.372215898s) [noise=true , address=45.91.116.13:4155 , piece_id=BJIO3BI6TOOJLAO7UWOWDEWS2YA7PHTD6XAQZBJ4R344R5FHXIQA, seg_pos={0 0}, upload=378516]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Dec 23 14:40:14.501169122 (431h11m14.372252167s) [noise=true , address=45.91.116.13:4155 , piece_id=BJIO3BI6TOOJLAO7UWOWDEWS2YA7PHTD6XAQZBJ4R344R5FHXIQA, seg_pos={0 0}, upload=378516]: Long tail cancellation due to optimal threshold Dec 23 14:40:14.501199750 (431h11m14.372282786s) [seg_pos={0 0}, upload=378516]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 14:40:14.501347652 (431h11m14.372430707s) [upload=378516]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 14:40:14.501356418 (431h11m14.372439463s) [upload=378516]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 14:40:14.797729085 (431h11m14.66881212s) [upload=378516]: Done uploading segments... Dec 23 14:40:14.797748543 (431h11m14.668831588s) [upload=378516]: Upload finished. err: <nil> Duration: 6.351839609s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [..........X..........X.XX..X............XX..X.............X.] duration: 6.001742803s trace: https://jaeger.dev.storj.io/trace/74788274d19d7a46 FAIL (2025-12-23 14:41:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA, Piece ID: PVTAKKVVQ427X5LMVEEJQVOO22NKBC6YRALCX3FYXOFJIBZVS3HQ) context deadline exceeded; piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: 2QPC7ELY3YBFYD7VUV7RHIPFY5PICWG5ZQMLSZRZKSD43Y6LBBOA) context deadline exceeded ======LOGS====== Dec 23 14:40:11.868119876 (431h11m11.739202911s) [upload=378516]: Starting upload Dec 23 14:40:11.868207653 (431h11m11.739290718s) [upload=378516]: Done starting upload Dec 23 14:40:11.868374521 (431h11m11.739457566s) [upload=378516]: Uploading segments... Dec 23 14:40:11.868506252 (431h11m11.739589297s) [upload=378516]: Got next segment. Inline: false Dec 23 14:40:11.868563651 (431h11m11.739646696s) [upload=378516]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 14:40:11.868569853 (431h11m11.739652898s) [upload=378516]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 14:40:12.111082779 (431h11m11.982165825s) [seg_pos={0 0}, upload=378516]: Begin upload segment... Dec 23 14:40:12.111134018 (431h11m11.982217053s) [seg_pos={0 0}, upload=378516]: Done begin upload segment. Dec 23 14:40:12.111151831 (431h11m11.982234866s) [noise=true , address=145.79.227.41:4085 , piece_id=PJ5H7E2LNKRAEAOFEWIVEV4USLM7DJ4XYD4GSQT5N5SQPWKLRLXA, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.111205434 (431h11m11.982288479s) [noise=true , address=77.90.31.15:4697 , piece_id=LRYC2NIG3MUVOMATXPEG4QMMIWKVED3HCBOQ6C7CXAJCSWVXQVTA, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.111360929 (431h11m11.982443974s) [noise=true , address=145.79.227.39:4013 , piece_id=ENCWF72DYPR54VZN4KXDT3U2CS54BJM2BQQ7BYA5ZP373FDWXZVA, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.111802781 (431h11m11.982885826s) [noise=true , address=77.90.31.13:4627 , piece_id=2QPC7ELY3YBFYD7VUV7RHIPFY5PICWG5ZQMLSZRZKSD43Y6LBBOA, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.112248239 (431h11m11.983331294s) [noise=true , address=45.91.116.13:4155 , piece_id=BJIO3BI6TOOJLAO7UWOWDEWS2YA7PHTD6XAQZBJ4R344R5FHXIQA, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.112648541 (431h11m11.983731586s) [noise=true , address=145.79.227.42:4123 , piece_id=P53WTGLAZA2SBX7MEIMYJABAI76T552NUANBWBNXCDAUNR426CSQ, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.113366068 (431h11m11.984449113s) [noise=true , address=45.91.116.5:4298 , piece_id=LNIEPYC6VOANJJAHWGIR54C2NSQNVBLW56WLJS3ATE3RMQI2WTQQ, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.113765128 (431h11m11.984848163s) [noise=true , address=45.91.116.9:4261 , piece_id=PZIQACPNQHHQD4MILGT6KPOOKU4D6O5FPG4D4TWQOVYLCGOPPAGQ, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.114301239 (431h11m11.985384304s) [noise=true , address=145.79.227.37:4565 , piece_id=K62BRPWTKPSD4IJD2NS2P7QC4JZ3QNLH42GO7XZKI2EAXF62K4KQ, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.114637028 (431h11m11.985720063s) [noise=true , address=45.91.116.12:4228 , piece_id=PVTAKKVVQ427X5LMVEEJQVOO22NKBC6YRALCX3FYXOFJIBZVS3HQ, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.115428274 (431h11m11.986511319s) [noise=true , address=145.79.227.49:4339 , piece_id=PQFUWEUSC7MOP6NNXIOE5M4DFA5OIVRLI3NW2DA6DATVCWQMUGJA, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.115717145 (431h11m11.98680018s) [noise=true , address=145.79.227.4:4187 , piece_id=LIXR5F3EXJTP6ALXC3A3V54IKIAK634JCBRONOCFN6RYI5KGOBHQ, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.116286669 (431h11m11.987369714s) [noise=true , address=145.79.227.50:4373 , piece_id=63RBNMEQ2IDHDV73J4FXZHB37KECBX6SWVNRLM36V5EPWKYTKYMQ, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.116830806 (431h11m11.987913841s) [noise=true , address=145.79.227.14:4839 , piece_id=KKNTSJISRHZKMTMDT75F2QDQF4EV6PTQ46QQTOLWFOGX5E5LDWOQ, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.117007933 (431h11m11.988090968s) [noise=true , address=145.79.227.45:4231 , piece_id=ZLYVEFQLJXGZ4PJ6HRXJZLPQWDWTOH7YAZTH3CCMVXUANT2DVGMA, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.117627111 (431h11m11.988710146s) [noise=true , address=145.79.227.11:4730 , piece_id=M5ZYH4B3UNIY65JJFGGRVXBR2VIOEVEJSK4AFO6SMBEFX2ZXYXYQ, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:13.362711746 (431h11m13.233794791s) [noise=true , address=77.90.31.15:4697 , piece_id=LRYC2NIG3MUVOMATXPEG4QMMIWKVED3HCBOQ6C7CXAJCSWVXQVTA, seg_pos={0 0}, upload=378516]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: storage node overloaded, request limit: 2000 Dec 23 14:40:13.824993954 (431h11m13.696076989s) [upload=378516]: Next returned nil segment Dec 23 14:40:13.825024922 (431h11m13.696107957s) [upload=378516]: Waiting for error group managing segments... Dec 23 14:40:14.102416555 (431h11m13.9734996s) [noise=true , address=145.79.227.41:4085 , piece_id=PJ5H7E2LNKRAEAOFEWIVEV4USLM7DJ4XYD4GSQT5N5SQPWKLRLXA, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.104229709 (431h11m13.975312744s) [noise=true , address=145.79.227.4:4187 , piece_id=LIXR5F3EXJTP6ALXC3A3V54IKIAK634JCBRONOCFN6RYI5KGOBHQ, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.108343191 (431h11m13.979426236s) [noise=true , address=145.79.227.50:4373 , piece_id=63RBNMEQ2IDHDV73J4FXZHB37KECBX6SWVNRLM36V5EPWKYTKYMQ, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.108974293 (431h11m13.980057338s) [noise=true , address=145.79.227.11:4730 , piece_id=M5ZYH4B3UNIY65JJFGGRVXBR2VIOEVEJSK4AFO6SMBEFX2ZXYXYQ, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.115937562 (431h11m13.987020607s) [noise=true , address=145.79.227.37:4565 , piece_id=K62BRPWTKPSD4IJD2NS2P7QC4JZ3QNLH42GO7XZKI2EAXF62K4KQ, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.124740384 (431h11m13.995823419s) [noise=true , address=145.79.227.39:4013 , piece_id=ENCWF72DYPR54VZN4KXDT3U2CS54BJM2BQQ7BYA5ZP373FDWXZVA, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.127790853 (431h11m13.998873898s) [noise=true , address=145.79.227.42:4123 , piece_id=P53WTGLAZA2SBX7MEIMYJABAI76T552NUANBWBNXCDAUNR426CSQ, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.185052439 (431h11m14.056135484s) [noise=true , address=145.79.227.49:4339 , piece_id=PQFUWEUSC7MOP6NNXIOE5M4DFA5OIVRLI3NW2DA6DATVCWQMUGJA, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.190499301 (431h11m14.061582346s) [noise=true , address=145.79.227.45:4231 , piece_id=ZLYVEFQLJXGZ4PJ6HRXJZLPQWDWTOH7YAZTH3CCMVXUANT2DVGMA, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.190648225 (431h11m14.06173127s) [noise=true , address=145.79.227.14:4839 , piece_id=KKNTSJISRHZKMTMDT75F2QDQF4EV6PTQ46QQTOLWFOGX5E5LDWOQ, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.382679737 (431h11m14.253762772s) [noise=true , address=45.91.116.9:4261 , piece_id=PZIQACPNQHHQD4MILGT6KPOOKU4D6O5FPG4D4TWQOVYLCGOPPAGQ, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.487075846 (431h11m14.358158881s) [noise=true , address=77.90.31.13:4627 , piece_id=2QPC7ELY3YBFYD7VUV7RHIPFY5PICWG5ZQMLSZRZKSD43Y6LBBOA, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.500579970 (431h11m14.371663005s) [noise=true , address=45.91.116.12:4228 , piece_id=PVTAKKVVQ427X5LMVEEJQVOO22NKBC6YRALCX3FYXOFJIBZVS3HQ, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.500606741 (431h11m14.371689786s) [seg_pos={0 0}, upload=378516]: Segment reached optimal threshold of 13 pieces. Dec 23 14:40:14.500868379 (431h11m14.371951424s) [noise=true , address=45.91.116.5:4298 , piece_id=LNIEPYC6VOANJJAHWGIR54C2NSQNVBLW56WLJS3ATE3RMQI2WTQQ, seg_pos={0 0}, upload=378516]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: write tcp 188.166.225.170:60010->45.91.116.5:4298: use of closed network connection; write tcp 188.166.225.170:60010->45.91.116.5:4298: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60010->45.91.116.5:4298: use of closed network connection Dec 23 14:40:14.500879801 (431h11m14.371962867s) [noise=true , address=45.91.116.5:4298 , piece_id=LNIEPYC6VOANJJAHWGIR54C2NSQNVBLW56WLJS3ATE3RMQI2WTQQ, seg_pos={0 0}, upload=378516]: Long tail cancellation due to optimal threshold Dec 23 14:40:14.501132863 (431h11m14.372215898s) [noise=true , address=45.91.116.13:4155 , piece_id=BJIO3BI6TOOJLAO7UWOWDEWS2YA7PHTD6XAQZBJ4R344R5FHXIQA, seg_pos={0 0}, upload=378516]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Dec 23 14:40:14.501169122 (431h11m14.372252167s) [noise=true , address=45.91.116.13:4155 , piece_id=BJIO3BI6TOOJLAO7UWOWDEWS2YA7PHTD6XAQZBJ4R344R5FHXIQA, seg_pos={0 0}, upload=378516]: Long tail cancellation due to optimal threshold Dec 23 14:40:14.501199750 (431h11m14.372282786s) [seg_pos={0 0}, upload=378516]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 14:40:14.501347652 (431h11m14.372430707s) [upload=378516]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 14:40:14.501356418 (431h11m14.372439463s) [upload=378516]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 14:40:14.797729085 (431h11m14.66881212s) [upload=378516]: Done uploading segments... Dec 23 14:40:14.797748543 (431h11m14.668831588s) [upload=378516]: Upload finished. err: <nil> Duration: 6.001774975s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [....X.....X............X....................................] duration: 3.898132493s trace: https://jaeger.dev.storj.io/trace/772f273c480c0b2b FAIL (2025-12-23 14:40:53 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.898159944s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [...........X..........X.XX..X............XX..X.............X] duration: 30.003251542s trace: https://jaeger.dev.storj.io/trace/74788274d19d7a46 FAIL (2025-12-23 14:40:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA, Piece ID: PVTAKKVVQ427X5LMVEEJQVOO22NKBC6YRALCX3FYXOFJIBZVS3HQ) context deadline exceeded; piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: 2QPC7ELY3YBFYD7VUV7RHIPFY5PICWG5ZQMLSZRZKSD43Y6LBBOA) context deadline exceeded ======LOGS====== Dec 23 14:40:11.868119876 (431h11m11.739202911s) [upload=378516]: Starting upload Dec 23 14:40:11.868207653 (431h11m11.739290718s) [upload=378516]: Done starting upload Dec 23 14:40:11.868374521 (431h11m11.739457566s) [upload=378516]: Uploading segments... Dec 23 14:40:11.868506252 (431h11m11.739589297s) [upload=378516]: Got next segment. Inline: false Dec 23 14:40:11.868563651 (431h11m11.739646696s) [upload=378516]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 14:40:11.868569853 (431h11m11.739652898s) [upload=378516]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 14:40:12.111082779 (431h11m11.982165825s) [seg_pos={0 0}, upload=378516]: Begin upload segment... Dec 23 14:40:12.111134018 (431h11m11.982217053s) [seg_pos={0 0}, upload=378516]: Done begin upload segment. Dec 23 14:40:12.111151831 (431h11m11.982234866s) [noise=true , address=145.79.227.41:4085 , piece_id=PJ5H7E2LNKRAEAOFEWIVEV4USLM7DJ4XYD4GSQT5N5SQPWKLRLXA, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.111205434 (431h11m11.982288479s) [noise=true , address=77.90.31.15:4697 , piece_id=LRYC2NIG3MUVOMATXPEG4QMMIWKVED3HCBOQ6C7CXAJCSWVXQVTA, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.111360929 (431h11m11.982443974s) [noise=true , address=145.79.227.39:4013 , piece_id=ENCWF72DYPR54VZN4KXDT3U2CS54BJM2BQQ7BYA5ZP373FDWXZVA, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.111802781 (431h11m11.982885826s) [noise=true , address=77.90.31.13:4627 , piece_id=2QPC7ELY3YBFYD7VUV7RHIPFY5PICWG5ZQMLSZRZKSD43Y6LBBOA, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.112248239 (431h11m11.983331294s) [noise=true , address=45.91.116.13:4155 , piece_id=BJIO3BI6TOOJLAO7UWOWDEWS2YA7PHTD6XAQZBJ4R344R5FHXIQA, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.112648541 (431h11m11.983731586s) [noise=true , address=145.79.227.42:4123 , piece_id=P53WTGLAZA2SBX7MEIMYJABAI76T552NUANBWBNXCDAUNR426CSQ, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.113366068 (431h11m11.984449113s) [noise=true , address=45.91.116.5:4298 , piece_id=LNIEPYC6VOANJJAHWGIR54C2NSQNVBLW56WLJS3ATE3RMQI2WTQQ, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.113765128 (431h11m11.984848163s) [noise=true , address=45.91.116.9:4261 , piece_id=PZIQACPNQHHQD4MILGT6KPOOKU4D6O5FPG4D4TWQOVYLCGOPPAGQ, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.114301239 (431h11m11.985384304s) [noise=true , address=145.79.227.37:4565 , piece_id=K62BRPWTKPSD4IJD2NS2P7QC4JZ3QNLH42GO7XZKI2EAXF62K4KQ, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.114637028 (431h11m11.985720063s) [noise=true , address=45.91.116.12:4228 , piece_id=PVTAKKVVQ427X5LMVEEJQVOO22NKBC6YRALCX3FYXOFJIBZVS3HQ, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.115428274 (431h11m11.986511319s) [noise=true , address=145.79.227.49:4339 , piece_id=PQFUWEUSC7MOP6NNXIOE5M4DFA5OIVRLI3NW2DA6DATVCWQMUGJA, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.115717145 (431h11m11.98680018s) [noise=true , address=145.79.227.4:4187 , piece_id=LIXR5F3EXJTP6ALXC3A3V54IKIAK634JCBRONOCFN6RYI5KGOBHQ, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.116286669 (431h11m11.987369714s) [noise=true , address=145.79.227.50:4373 , piece_id=63RBNMEQ2IDHDV73J4FXZHB37KECBX6SWVNRLM36V5EPWKYTKYMQ, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.116830806 (431h11m11.987913841s) [noise=true , address=145.79.227.14:4839 , piece_id=KKNTSJISRHZKMTMDT75F2QDQF4EV6PTQ46QQTOLWFOGX5E5LDWOQ, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.117007933 (431h11m11.988090968s) [noise=true , address=145.79.227.45:4231 , piece_id=ZLYVEFQLJXGZ4PJ6HRXJZLPQWDWTOH7YAZTH3CCMVXUANT2DVGMA, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.117627111 (431h11m11.988710146s) [noise=true , address=145.79.227.11:4730 , piece_id=M5ZYH4B3UNIY65JJFGGRVXBR2VIOEVEJSK4AFO6SMBEFX2ZXYXYQ, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:13.362711746 (431h11m13.233794791s) [noise=true , address=77.90.31.15:4697 , piece_id=LRYC2NIG3MUVOMATXPEG4QMMIWKVED3HCBOQ6C7CXAJCSWVXQVTA, seg_pos={0 0}, upload=378516]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: storage node overloaded, request limit: 2000 Dec 23 14:40:13.824993954 (431h11m13.696076989s) [upload=378516]: Next returned nil segment Dec 23 14:40:13.825024922 (431h11m13.696107957s) [upload=378516]: Waiting for error group managing segments... Dec 23 14:40:14.102416555 (431h11m13.9734996s) [noise=true , address=145.79.227.41:4085 , piece_id=PJ5H7E2LNKRAEAOFEWIVEV4USLM7DJ4XYD4GSQT5N5SQPWKLRLXA, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.104229709 (431h11m13.975312744s) [noise=true , address=145.79.227.4:4187 , piece_id=LIXR5F3EXJTP6ALXC3A3V54IKIAK634JCBRONOCFN6RYI5KGOBHQ, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.108343191 (431h11m13.979426236s) [noise=true , address=145.79.227.50:4373 , piece_id=63RBNMEQ2IDHDV73J4FXZHB37KECBX6SWVNRLM36V5EPWKYTKYMQ, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.108974293 (431h11m13.980057338s) [noise=true , address=145.79.227.11:4730 , piece_id=M5ZYH4B3UNIY65JJFGGRVXBR2VIOEVEJSK4AFO6SMBEFX2ZXYXYQ, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.115937562 (431h11m13.987020607s) [noise=true , address=145.79.227.37:4565 , piece_id=K62BRPWTKPSD4IJD2NS2P7QC4JZ3QNLH42GO7XZKI2EAXF62K4KQ, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.124740384 (431h11m13.995823419s) [noise=true , address=145.79.227.39:4013 , piece_id=ENCWF72DYPR54VZN4KXDT3U2CS54BJM2BQQ7BYA5ZP373FDWXZVA, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.127790853 (431h11m13.998873898s) [noise=true , address=145.79.227.42:4123 , piece_id=P53WTGLAZA2SBX7MEIMYJABAI76T552NUANBWBNXCDAUNR426CSQ, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.185052439 (431h11m14.056135484s) [noise=true , address=145.79.227.49:4339 , piece_id=PQFUWEUSC7MOP6NNXIOE5M4DFA5OIVRLI3NW2DA6DATVCWQMUGJA, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.190499301 (431h11m14.061582346s) [noise=true , address=145.79.227.45:4231 , piece_id=ZLYVEFQLJXGZ4PJ6HRXJZLPQWDWTOH7YAZTH3CCMVXUANT2DVGMA, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.190648225 (431h11m14.06173127s) [noise=true , address=145.79.227.14:4839 , piece_id=KKNTSJISRHZKMTMDT75F2QDQF4EV6PTQ46QQTOLWFOGX5E5LDWOQ, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.382679737 (431h11m14.253762772s) [noise=true , address=45.91.116.9:4261 , piece_id=PZIQACPNQHHQD4MILGT6KPOOKU4D6O5FPG4D4TWQOVYLCGOPPAGQ, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.487075846 (431h11m14.358158881s) [noise=true , address=77.90.31.13:4627 , piece_id=2QPC7ELY3YBFYD7VUV7RHIPFY5PICWG5ZQMLSZRZKSD43Y6LBBOA, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.500579970 (431h11m14.371663005s) [noise=true , address=45.91.116.12:4228 , piece_id=PVTAKKVVQ427X5LMVEEJQVOO22NKBC6YRALCX3FYXOFJIBZVS3HQ, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.500606741 (431h11m14.371689786s) [seg_pos={0 0}, upload=378516]: Segment reached optimal threshold of 13 pieces. Dec 23 14:40:14.500868379 (431h11m14.371951424s) [noise=true , address=45.91.116.5:4298 , piece_id=LNIEPYC6VOANJJAHWGIR54C2NSQNVBLW56WLJS3ATE3RMQI2WTQQ, seg_pos={0 0}, upload=378516]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: write tcp 188.166.225.170:60010->45.91.116.5:4298: use of closed network connection; write tcp 188.166.225.170:60010->45.91.116.5:4298: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60010->45.91.116.5:4298: use of closed network connection Dec 23 14:40:14.500879801 (431h11m14.371962867s) [noise=true , address=45.91.116.5:4298 , piece_id=LNIEPYC6VOANJJAHWGIR54C2NSQNVBLW56WLJS3ATE3RMQI2WTQQ, seg_pos={0 0}, upload=378516]: Long tail cancellation due to optimal threshold Dec 23 14:40:14.501132863 (431h11m14.372215898s) [noise=true , address=45.91.116.13:4155 , piece_id=BJIO3BI6TOOJLAO7UWOWDEWS2YA7PHTD6XAQZBJ4R344R5FHXIQA, seg_pos={0 0}, upload=378516]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Dec 23 14:40:14.501169122 (431h11m14.372252167s) [noise=true , address=45.91.116.13:4155 , piece_id=BJIO3BI6TOOJLAO7UWOWDEWS2YA7PHTD6XAQZBJ4R344R5FHXIQA, seg_pos={0 0}, upload=378516]: Long tail cancellation due to optimal threshold Dec 23 14:40:14.501199750 (431h11m14.372282786s) [seg_pos={0 0}, upload=378516]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 14:40:14.501347652 (431h11m14.372430707s) [upload=378516]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 14:40:14.501356418 (431h11m14.372439463s) [upload=378516]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 14:40:14.797729085 (431h11m14.66881212s) [upload=378516]: Done uploading segments... Dec 23 14:40:14.797748543 (431h11m14.668831588s) [upload=378516]: Upload finished. err: <nil> Duration: 30.003273735s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [............X..........X.XX..X............XX..X.............] duration: 7.47083035s trace: https://jaeger.dev.storj.io/trace/215296e6cb5de778 FAIL (2025-12-23 14:39:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9, Piece ID: TGN5OFP66O6EI654N63FVJSCOXFSIS3AURJGAGYWLSBHVMAOFYFQ) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: 3MBFI2ILY6VP6XT2PMXFVWG7UA2HYOYYNZASSLX3MLQWTEV7EAKA) context deadline exceeded ======LOGS====== Dec 23 14:28:02.333620097 (430h59m2.204703152s) [upload=378343]: Starting upload Dec 23 14:28:02.333737752 (430h59m2.204820797s) [upload=378343]: Done starting upload Dec 23 14:28:02.333838052 (430h59m2.204921107s) [upload=378343]: Uploading segments... Dec 23 14:28:02.334165316 (430h59m2.205248381s) [upload=378343]: Got next segment. Inline: false Dec 23 14:28:02.334217274 (430h59m2.205300319s) [upload=378343]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 14:28:02.334693501 (430h59m2.205776567s) [upload=378343]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 14:28:02.588168613 (430h59m2.459251658s) [seg_pos={0 0}, upload=378343]: Begin upload segment... Dec 23 14:28:02.588224871 (430h59m2.459307916s) [seg_pos={0 0}, upload=378343]: Done begin upload segment. Dec 23 14:28:02.588264035 (430h59m2.45934707s) [noise=true , address=145.79.227.32:4834 , piece_id=FLPDIEXB34RV3VYR4LZYW63M76C3G2352VTJPF6IHDJSKAY2VSIQ, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.588315313 (430h59m2.459398349s) [noise=true , address=145.79.227.11:4731 , piece_id=JDCFIGBBNHW67SQCYI7PKCKWPUQBCGO3BZROXPHNABIGFFMATMOQ, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.589826190 (430h59m2.460909225s) [noise=true , address=145.79.227.37:4565 , piece_id=RIXR6277D5GYIDPGNMMSJAQLFCMM4P6LWALPUSNFW67BWRXQJ4SQ, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.589974662 (430h59m2.461057708s) [noise=true , address=145.79.227.50:4375 , piece_id=2L6LRC33345CFQ776H4C3RBHD27J3WN5BBB7QUEQ43EQED5LQJRA, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.590200293 (430h59m2.461283328s) [noise=true , address=145.79.227.52:4445 , piece_id=VIE24CGHGMPQSCGUQKOGNIOMJMZHYD4BRZSGEPMUSPPAXPSFOHZQ, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.590571109 (430h59m2.461654154s) [noise=true , address=145.79.227.39:4014 , piece_id=UIYHVARR7QJN7TC45DPH7YMWLOV27EVHFR4HNZXTNGCFCYZ5QWSQ, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.590591117 (430h59m2.461674162s) [noise=true , address=77.90.31.14:4664 , piece_id=TGN5OFP66O6EI654N63FVJSCOXFSIS3AURJGAGYWLSBHVMAOFYFQ, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.591120444 (430h59m2.462203489s) [noise=true , address=145.79.227.38:4978 , piece_id=7M2D3NR342SZVWHS754YCZYVPX2SAKZDUHNV5MIUMG7NQYMJORXA, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.591482514 (430h59m2.462565559s) [noise=true , address=145.79.227.15:4873 , piece_id=RROMZU3GQXWHZ2KAHKJNUXB6JYJW5DOZIMS4RTI2CK4H6GJ4JJPA, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.592004899 (430h59m2.463087944s) [noise=true , address=45.91.116.6:4975 , piece_id=3MBFI2ILY6VP6XT2PMXFVWG7UA2HYOYYNZASSLX3MLQWTEV7EAKA, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.592099669 (430h59m2.463182724s) [noise=true , address=145.79.227.20:4438 , piece_id=OHBU6DZOK7H6MIDZHMXEPL2MDY2OXQSX3VPDIE3RDW2THSMC2IMA, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.593944512 (430h59m2.465027557s) [noise=true , address=145.79.227.16:4910 , piece_id=FDXOMBKKQQPPL3G577PWWCWDJFSGZ3W3Q6OZYANVQ7XSAINJ7MFA, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.597912307 (430h59m2.468995352s) [noise=true , address=145.79.227.29:4943 , piece_id=FF4Q6CY7O4EUFQO5QY7GHSYAAQFHK2WCH7A2BR5B6JGJH4MGHSTA, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.600520093 (430h59m2.471603128s) [noise=true , address=145.79.227.40:4049 , piece_id=3QAECV5EQI3BBHUW7AZXVYWZY6DAOOH6DVP4XAMDXBFJAV34QB5Q, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.602548155 (430h59m2.47363121s) [noise=true , address=45.91.116.4:4939 , piece_id=3BWKNBSDTQ4DBJFYXFY3IPWOJ3JTPPDMRO2DRYA5REJ3BERVMFKQ, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.604654816 (430h59m2.475737861s) [noise=true , address=145.79.227.51:4410 , piece_id=T5XXHOFBLW3A3OKTDY5CTO74VFCY5ZQJRJPACYF3SCLOTF6E25LA, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:04.341061516 (430h59m4.212144561s) [upload=378343]: Next returned nil segment Dec 23 14:28:04.341149894 (430h59m4.212232949s) [upload=378343]: Waiting for error group managing segments... Dec 23 14:28:04.631523518 (430h59m4.502606563s) [noise=true , address=145.79.227.16:4910 , piece_id=FDXOMBKKQQPPL3G577PWWCWDJFSGZ3W3Q6OZYANVQ7XSAINJ7MFA, seg_pos={0 0}, upload=378343]: Done uploading piece. err: <nil> Dec 23 14:28:04.649224493 (430h59m4.520307538s) [noise=true , address=145.79.227.29:4943 , piece_id=FF4Q6CY7O4EUFQO5QY7GHSYAAQFHK2WCH7A2BR5B6JGJH4MGHSTA, seg_pos={0 0}, upload=378343]: Done uploading piece. err: <nil> Dec 23 14:28:04.655099941 (430h59m4.526182986s) [noise=true , address=145.79.227.38:4978 , piece_id=7M2D3NR342SZVWHS754YCZYVPX2SAKZDUHNV5MIUMG7NQYMJORXA, seg_pos={0 0}, upload=378343]: Done uploading piece. err: <nil> Dec 23 14:28:04.663569098 (430h59m4.534652133s) [noise=true , address=145.79.227.15:4873 , piece_id=RROMZU3GQXWHZ2KAHKJNUXB6JYJW5DOZIMS4RTI2CK4H6GJ4JJPA, seg_pos={0 0}, upload=378343]: Done uploading piece. err: <nil> Dec 23 14:28:04.784039696 (430h59m4.655122741s) [noise=true , address=145.79.227.39:4014 , piece_id=UIYHVARR7QJN7TC45DPH7YMWLOV27EVHFR4HNZXTNGCFCYZ5QWSQ, seg_pos={0 0}, upload=378343]: Done uploading piece. err: <nil> Dec 23 14:28:04.789452723 (430h59m4.660535769s) [noise=true , address=145.79.227.50:4375 , piece_id=2L6LRC33345CFQ776H4C3RBHD27J3WN5BBB7QUEQ43EQED5LQJRA, seg_pos={0 0}, upload=378343]: Done uploading piece. err: <nil> Dec 23 14:28:04.809692453 (430h59m4.680775488s) [noise=true , address=145.79.227.32:4834 , piece_id=FLPDIEXB34RV3VYR4LZYW63M76C3G2352VTJPF6IHDJSKAY2VSIQ, seg_pos={0 0}, upload=378343]: Done uploading piece. err: <nil> Dec 23 14:28:04.814157645 (430h59m4.68524069s) [noise=true , address=145.79.227.20:4438 , piece_id=OHBU6DZOK7H6MIDZHMXEPL2MDY2OXQSX3VPDIE3RDW2THSMC2IMA, seg_pos={0 0}, upload=378343]: Done uploading piece. err: <nil> Dec 23 14:28:04.831851378 (430h59m4.702934423s) [noise=true , address=45.91.116.6:4975 , piece_id=3MBFI2ILY6VP6XT2PMXFVWG7UA2HYOYYNZASSLX3MLQWTEV7EAKA, seg_pos={0 0}, upload=378343]: Done uploading piece. err: <nil> Dec 23 14:28:04.884609870 (430h59m4.755692915s) [noise=true , address=45.91.116.4:4939 , piece_id=3BWKNBSDTQ4DBJFYXFY3IPWOJ3JTPPDMRO2DRYA5REJ3BERVMFKQ, seg_pos={0 0}, upload=378343]: Done uploading piece. err: <nil> Dec 23 14:28:05.131970658 (430h59m5.003053693s) [noise=true , address=77.90.31.14:4664 , piece_id=TGN5OFP66O6EI654N63FVJSCOXFSIS3AURJGAGYWLSBHVMAOFYFQ, seg_pos={0 0}, upload=378343]: Done uploading piece. err: <nil> Dec 23 14:28:05.195398928 (430h59m5.066481983s) [noise=true , address=145.79.227.51:4410 , piece_id=T5XXHOFBLW3A3OKTDY5CTO74VFCY5ZQJRJPACYF3SCLOTF6E25LA, seg_pos={0 0}, upload=378343]: Done uploading piece. err: <nil> Dec 23 14:28:05.218188272 (430h59m5.089271317s) [noise=true , address=145.79.227.11:4731 , piece_id=JDCFIGBBNHW67SQCYI7PKCKWPUQBCGO3BZROXPHNABIGFFMATMOQ, seg_pos={0 0}, upload=378343]: Done uploading piece. err: <nil> Dec 23 14:28:05.218214873 (430h59m5.089297908s) [seg_pos={0 0}, upload=378343]: Segment reached optimal threshold of 13 pieces. Dec 23 14:28:05.218460901 (430h59m5.089543966s) [noise=true , address=145.79.227.37:4565 , piece_id=RIXR6277D5GYIDPGNMMSJAQLFCMM4P6LWALPUSNFW67BWRXQJ4SQ, seg_pos={0 0}, upload=378343]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nvvNczHtQrSEJuJbErnjiAWa4BRDzBrQmBNbB7RrrqN1JWNHT): protocol: write tcp 188.166.225.170:39840->145.79.227.37:4565: use of closed network connection; write tcp 188.166.225.170:39840->145.79.227.37:4565: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39840->145.79.227.37:4565: use of closed network connection Dec 23 14:28:05.218469708 (430h59m5.089552753s) [noise=true , address=145.79.227.37:4565 , piece_id=RIXR6277D5GYIDPGNMMSJAQLFCMM4P6LWALPUSNFW67BWRXQJ4SQ, seg_pos={0 0}, upload=378343]: Long tail cancellation due to optimal threshold Dec 23 14:28:05.218750874 (430h59m5.089833919s) [noise=true , address=145.79.227.40:4049 , piece_id=3QAECV5EQI3BBHUW7AZXVYWZY6DAOOH6DVP4XAMDXBFJAV34QB5Q, seg_pos={0 0}, upload=378343]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF Dec 23 14:28:05.218770391 (430h59m5.089853436s) [noise=true , address=145.79.227.40:4049 , piece_id=3QAECV5EQI3BBHUW7AZXVYWZY6DAOOH6DVP4XAMDXBFJAV34QB5Q, seg_pos={0 0}, upload=378343]: Long tail cancellation due to optimal threshold Dec 23 14:28:05.219134284 (430h59m5.090217349s) [noise=true , address=145.79.227.52:4445 , piece_id=VIE24CGHGMPQSCGUQKOGNIOMJMZHYD4BRZSGEPMUSPPAXPSFOHZQ, seg_pos={0 0}, upload=378343]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12vXrkhLqCofhR4Fmfqb98gxnqWQF6vMV4q3ATbRvPfcqi82WSo): protocol: expected piece hash; context canceled; EOF Dec 23 14:28:05.219146146 (430h59m5.090229201s) [noise=true , address=145.79.227.52:4445 , piece_id=VIE24CGHGMPQSCGUQKOGNIOMJMZHYD4BRZSGEPMUSPPAXPSFOHZQ, seg_pos={0 0}, upload=378343]: Long tail cancellation due to optimal threshold Dec 23 14:28:05.219173128 (430h59m5.090256173s) [seg_pos={0 0}, upload=378343]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 14:28:05.219417493 (430h59m5.090500538s) [upload=378343]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 14:28:05.219425228 (430h59m5.090508293s) [upload=378343]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 14:28:05.522079636 (430h59m5.393162671s) [upload=378343]: Done uploading segments... Dec 23 14:28:05.522100636 (430h59m5.393183681s) [upload=378343]: Upload finished. err: <nil> Duration: 7.470856179s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.....X.....X............X...................................] duration: 3.807157949s trace: https://jaeger.dev.storj.io/trace/772f273c480c0b2b FAIL (2025-12-23 14:39:49 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.807191423s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [......X.....X............X..................................] duration: 3.638071259s trace: https://jaeger.dev.storj.io/trace/772f273c480c0b2b FAIL (2025-12-23 14:39:04 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.638099452s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.............X..........X.XX..X............XX..X............] duration: 8.536761743s trace: https://jaeger.dev.storj.io/trace/215296e6cb5de778 FAIL (2025-12-23 14:38:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9, Piece ID: TGN5OFP66O6EI654N63FVJSCOXFSIS3AURJGAGYWLSBHVMAOFYFQ) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: 3MBFI2ILY6VP6XT2PMXFVWG7UA2HYOYYNZASSLX3MLQWTEV7EAKA) context deadline exceeded ======LOGS====== Dec 23 14:28:02.333620097 (430h59m2.204703152s) [upload=378343]: Starting upload Dec 23 14:28:02.333737752 (430h59m2.204820797s) [upload=378343]: Done starting upload Dec 23 14:28:02.333838052 (430h59m2.204921107s) [upload=378343]: Uploading segments... Dec 23 14:28:02.334165316 (430h59m2.205248381s) [upload=378343]: Got next segment. Inline: false Dec 23 14:28:02.334217274 (430h59m2.205300319s) [upload=378343]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 14:28:02.334693501 (430h59m2.205776567s) [upload=378343]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 14:28:02.588168613 (430h59m2.459251658s) [seg_pos={0 0}, upload=378343]: Begin upload segment... Dec 23 14:28:02.588224871 (430h59m2.459307916s) [seg_pos={0 0}, upload=378343]: Done begin upload segment. Dec 23 14:28:02.588264035 (430h59m2.45934707s) [noise=true , address=145.79.227.32:4834 , piece_id=FLPDIEXB34RV3VYR4LZYW63M76C3G2352VTJPF6IHDJSKAY2VSIQ, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.588315313 (430h59m2.459398349s) [noise=true , address=145.79.227.11:4731 , piece_id=JDCFIGBBNHW67SQCYI7PKCKWPUQBCGO3BZROXPHNABIGFFMATMOQ, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.589826190 (430h59m2.460909225s) [noise=true , address=145.79.227.37:4565 , piece_id=RIXR6277D5GYIDPGNMMSJAQLFCMM4P6LWALPUSNFW67BWRXQJ4SQ, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.589974662 (430h59m2.461057708s) [noise=true , address=145.79.227.50:4375 , piece_id=2L6LRC33345CFQ776H4C3RBHD27J3WN5BBB7QUEQ43EQED5LQJRA, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.590200293 (430h59m2.461283328s) [noise=true , address=145.79.227.52:4445 , piece_id=VIE24CGHGMPQSCGUQKOGNIOMJMZHYD4BRZSGEPMUSPPAXPSFOHZQ, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.590571109 (430h59m2.461654154s) [noise=true , address=145.79.227.39:4014 , piece_id=UIYHVARR7QJN7TC45DPH7YMWLOV27EVHFR4HNZXTNGCFCYZ5QWSQ, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.590591117 (430h59m2.461674162s) [noise=true , address=77.90.31.14:4664 , piece_id=TGN5OFP66O6EI654N63FVJSCOXFSIS3AURJGAGYWLSBHVMAOFYFQ, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.591120444 (430h59m2.462203489s) [noise=true , address=145.79.227.38:4978 , piece_id=7M2D3NR342SZVWHS754YCZYVPX2SAKZDUHNV5MIUMG7NQYMJORXA, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.591482514 (430h59m2.462565559s) [noise=true , address=145.79.227.15:4873 , piece_id=RROMZU3GQXWHZ2KAHKJNUXB6JYJW5DOZIMS4RTI2CK4H6GJ4JJPA, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.592004899 (430h59m2.463087944s) [noise=true , address=45.91.116.6:4975 , piece_id=3MBFI2ILY6VP6XT2PMXFVWG7UA2HYOYYNZASSLX3MLQWTEV7EAKA, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.592099669 (430h59m2.463182724s) [noise=true , address=145.79.227.20:4438 , piece_id=OHBU6DZOK7H6MIDZHMXEPL2MDY2OXQSX3VPDIE3RDW2THSMC2IMA, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.593944512 (430h59m2.465027557s) [noise=true , address=145.79.227.16:4910 , piece_id=FDXOMBKKQQPPL3G577PWWCWDJFSGZ3W3Q6OZYANVQ7XSAINJ7MFA, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.597912307 (430h59m2.468995352s) [noise=true , address=145.79.227.29:4943 , piece_id=FF4Q6CY7O4EUFQO5QY7GHSYAAQFHK2WCH7A2BR5B6JGJH4MGHSTA, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.600520093 (430h59m2.471603128s) [noise=true , address=145.79.227.40:4049 , piece_id=3QAECV5EQI3BBHUW7AZXVYWZY6DAOOH6DVP4XAMDXBFJAV34QB5Q, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.602548155 (430h59m2.47363121s) [noise=true , address=45.91.116.4:4939 , piece_id=3BWKNBSDTQ4DBJFYXFY3IPWOJ3JTPPDMRO2DRYA5REJ3BERVMFKQ, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.604654816 (430h59m2.475737861s) [noise=true , address=145.79.227.51:4410 , piece_id=T5XXHOFBLW3A3OKTDY5CTO74VFCY5ZQJRJPACYF3SCLOTF6E25LA, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:04.341061516 (430h59m4.212144561s) [upload=378343]: Next returned nil segment Dec 23 14:28:04.341149894 (430h59m4.212232949s) [upload=378343]: Waiting for error group managing segments... Dec 23 14:28:04.631523518 (430h59m4.502606563s) [noise=true , address=145.79.227.16:4910 , piece_id=FDXOMBKKQQPPL3G577PWWCWDJFSGZ3W3Q6OZYANVQ7XSAINJ7MFA, seg_pos={0 0}, upload=378343]: Done uploading piece. err: <nil> Dec 23 14:28:04.649224493 (430h59m4.520307538s) [noise=true , address=145.79.227.29:4943 , piece_id=FF4Q6CY7O4EUFQO5QY7GHSYAAQFHK2WCH7A2BR5B6JGJH4MGHSTA, seg_pos={0 0}, upload=378343]: Done uploading piece. err: <nil> Dec 23 14:28:04.655099941 (430h59m4.526182986s) [noise=true , address=145.79.227.38:4978 , piece_id=7M2D3NR342SZVWHS754YCZYVPX2SAKZDUHNV5MIUMG7NQYMJORXA, seg_pos={0 0}, upload=378343]: Done uploading piece. err: <nil> Dec 23 14:28:04.663569098 (430h59m4.534652133s) [noise=true , address=145.79.227.15:4873 , piece_id=RROMZU3GQXWHZ2KAHKJNUXB6JYJW5DOZIMS4RTI2CK4H6GJ4JJPA, seg_pos={0 0}, upload=378343]: Done uploading piece. err: <nil> Dec 23 14:28:04.784039696 (430h59m4.655122741s) [noise=true , address=145.79.227.39:4014 , piece_id=UIYHVARR7QJN7TC45DPH7YMWLOV27EVHFR4HNZXTNGCFCYZ5QWSQ, seg_pos={0 0}, upload=378343]: Done uploading piece. err: <nil> Dec 23 14:28:04.789452723 (430h59m4.660535769s) [noise=true , address=145.79.227.50:4375 , piece_id=2L6LRC33345CFQ776H4C3RBHD27J3WN5BBB7QUEQ43EQED5LQJRA, seg_pos={0 0}, upload=378343]: Done uploading piece. err: <nil> Dec 23 14:28:04.809692453 (430h59m4.680775488s) [noise=true , address=145.79.227.32:4834 , piece_id=FLPDIEXB34RV3VYR4LZYW63M76C3G2352VTJPF6IHDJSKAY2VSIQ, seg_pos={0 0}, upload=378343]: Done uploading piece. err: <nil> Dec 23 14:28:04.814157645 (430h59m4.68524069s) [noise=true , address=145.79.227.20:4438 , piece_id=OHBU6DZOK7H6MIDZHMXEPL2MDY2OXQSX3VPDIE3RDW2THSMC2IMA, seg_pos={0 0}, upload=378343]: Done uploading piece. err: <nil> Dec 23 14:28:04.831851378 (430h59m4.702934423s) [noise=true , address=45.91.116.6:4975 , piece_id=3MBFI2ILY6VP6XT2PMXFVWG7UA2HYOYYNZASSLX3MLQWTEV7EAKA, seg_pos={0 0}, upload=378343]: Done uploading piece. err: <nil> Dec 23 14:28:04.884609870 (430h59m4.755692915s) [noise=true , address=45.91.116.4:4939 , piece_id=3BWKNBSDTQ4DBJFYXFY3IPWOJ3JTPPDMRO2DRYA5REJ3BERVMFKQ, seg_pos={0 0}, upload=378343]: Done uploading piece. err: <nil> Dec 23 14:28:05.131970658 (430h59m5.003053693s) [noise=true , address=77.90.31.14:4664 , piece_id=TGN5OFP66O6EI654N63FVJSCOXFSIS3AURJGAGYWLSBHVMAOFYFQ, seg_pos={0 0}, upload=378343]: Done uploading piece. err: <nil> Dec 23 14:28:05.195398928 (430h59m5.066481983s) [noise=true , address=145.79.227.51:4410 , piece_id=T5XXHOFBLW3A3OKTDY5CTO74VFCY5ZQJRJPACYF3SCLOTF6E25LA, seg_pos={0 0}, upload=378343]: Done uploading piece. err: <nil> Dec 23 14:28:05.218188272 (430h59m5.089271317s) [noise=true , address=145.79.227.11:4731 , piece_id=JDCFIGBBNHW67SQCYI7PKCKWPUQBCGO3BZROXPHNABIGFFMATMOQ, seg_pos={0 0}, upload=378343]: Done uploading piece. err: <nil> Dec 23 14:28:05.218214873 (430h59m5.089297908s) [seg_pos={0 0}, upload=378343]: Segment reached optimal threshold of 13 pieces. Dec 23 14:28:05.218460901 (430h59m5.089543966s) [noise=true , address=145.79.227.37:4565 , piece_id=RIXR6277D5GYIDPGNMMSJAQLFCMM4P6LWALPUSNFW67BWRXQJ4SQ, seg_pos={0 0}, upload=378343]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nvvNczHtQrSEJuJbErnjiAWa4BRDzBrQmBNbB7RrrqN1JWNHT): protocol: write tcp 188.166.225.170:39840->145.79.227.37:4565: use of closed network connection; write tcp 188.166.225.170:39840->145.79.227.37:4565: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39840->145.79.227.37:4565: use of closed network connection Dec 23 14:28:05.218469708 (430h59m5.089552753s) [noise=true , address=145.79.227.37:4565 , piece_id=RIXR6277D5GYIDPGNMMSJAQLFCMM4P6LWALPUSNFW67BWRXQJ4SQ, seg_pos={0 0}, upload=378343]: Long tail cancellation due to optimal threshold Dec 23 14:28:05.218750874 (430h59m5.089833919s) [noise=true , address=145.79.227.40:4049 , piece_id=3QAECV5EQI3BBHUW7AZXVYWZY6DAOOH6DVP4XAMDXBFJAV34QB5Q, seg_pos={0 0}, upload=378343]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF Dec 23 14:28:05.218770391 (430h59m5.089853436s) [noise=true , address=145.79.227.40:4049 , piece_id=3QAECV5EQI3BBHUW7AZXVYWZY6DAOOH6DVP4XAMDXBFJAV34QB5Q, seg_pos={0 0}, upload=378343]: Long tail cancellation due to optimal threshold Dec 23 14:28:05.219134284 (430h59m5.090217349s) [noise=true , address=145.79.227.52:4445 , piece_id=VIE24CGHGMPQSCGUQKOGNIOMJMZHYD4BRZSGEPMUSPPAXPSFOHZQ, seg_pos={0 0}, upload=378343]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12vXrkhLqCofhR4Fmfqb98gxnqWQF6vMV4q3ATbRvPfcqi82WSo): protocol: expected piece hash; context canceled; EOF Dec 23 14:28:05.219146146 (430h59m5.090229201s) [noise=true , address=145.79.227.52:4445 , piece_id=VIE24CGHGMPQSCGUQKOGNIOMJMZHYD4BRZSGEPMUSPPAXPSFOHZQ, seg_pos={0 0}, upload=378343]: Long tail cancellation due to optimal threshold Dec 23 14:28:05.219173128 (430h59m5.090256173s) [seg_pos={0 0}, upload=378343]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 14:28:05.219417493 (430h59m5.090500538s) [upload=378343]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 14:28:05.219425228 (430h59m5.090508293s) [upload=378343]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 14:28:05.522079636 (430h59m5.393162671s) [upload=378343]: Done uploading segments... Dec 23 14:28:05.522100636 (430h59m5.393183681s) [upload=378343]: Upload finished. err: <nil> Duration: 8.53678638s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.......X.....X............X.................................] duration: 3.667100824s trace: https://jaeger.dev.storj.io/trace/772f273c480c0b2b FAIL (2025-12-23 14:38:22 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.667126874s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [..............X..........X.XX..X............XX..X...........] duration: 7.929764434s trace: https://jaeger.dev.storj.io/trace/215296e6cb5de778 FAIL (2025-12-23 14:37:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9, Piece ID: TGN5OFP66O6EI654N63FVJSCOXFSIS3AURJGAGYWLSBHVMAOFYFQ) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: 3MBFI2ILY6VP6XT2PMXFVWG7UA2HYOYYNZASSLX3MLQWTEV7EAKA) context deadline exceeded ======LOGS====== Dec 23 14:28:02.333620097 (430h59m2.204703152s) [upload=378343]: Starting upload Dec 23 14:28:02.333737752 (430h59m2.204820797s) [upload=378343]: Done starting upload Dec 23 14:28:02.333838052 (430h59m2.204921107s) [upload=378343]: Uploading segments... Dec 23 14:28:02.334165316 (430h59m2.205248381s) [upload=378343]: Got next segment. Inline: false Dec 23 14:28:02.334217274 (430h59m2.205300319s) [upload=378343]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 14:28:02.334693501 (430h59m2.205776567s) [upload=378343]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 14:28:02.588168613 (430h59m2.459251658s) [seg_pos={0 0}, upload=378343]: Begin upload segment... Dec 23 14:28:02.588224871 (430h59m2.459307916s) [seg_pos={0 0}, upload=378343]: Done begin upload segment. Dec 23 14:28:02.588264035 (430h59m2.45934707s) [noise=true , address=145.79.227.32:4834 , piece_id=FLPDIEXB34RV3VYR4LZYW63M76C3G2352VTJPF6IHDJSKAY2VSIQ, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.588315313 (430h59m2.459398349s) [noise=true , address=145.79.227.11:4731 , piece_id=JDCFIGBBNHW67SQCYI7PKCKWPUQBCGO3BZROXPHNABIGFFMATMOQ, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.589826190 (430h59m2.460909225s) [noise=true , address=145.79.227.37:4565 , piece_id=RIXR6277D5GYIDPGNMMSJAQLFCMM4P6LWALPUSNFW67BWRXQJ4SQ, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.589974662 (430h59m2.461057708s) [noise=true , address=145.79.227.50:4375 , piece_id=2L6LRC33345CFQ776H4C3RBHD27J3WN5BBB7QUEQ43EQED5LQJRA, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.590200293 (430h59m2.461283328s) [noise=true , address=145.79.227.52:4445 , piece_id=VIE24CGHGMPQSCGUQKOGNIOMJMZHYD4BRZSGEPMUSPPAXPSFOHZQ, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.590571109 (430h59m2.461654154s) [noise=true , address=145.79.227.39:4014 , piece_id=UIYHVARR7QJN7TC45DPH7YMWLOV27EVHFR4HNZXTNGCFCYZ5QWSQ, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.590591117 (430h59m2.461674162s) [noise=true , address=77.90.31.14:4664 , piece_id=TGN5OFP66O6EI654N63FVJSCOXFSIS3AURJGAGYWLSBHVMAOFYFQ, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.591120444 (430h59m2.462203489s) [noise=true , address=145.79.227.38:4978 , piece_id=7M2D3NR342SZVWHS754YCZYVPX2SAKZDUHNV5MIUMG7NQYMJORXA, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.591482514 (430h59m2.462565559s) [noise=true , address=145.79.227.15:4873 , piece_id=RROMZU3GQXWHZ2KAHKJNUXB6JYJW5DOZIMS4RTI2CK4H6GJ4JJPA, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.592004899 (430h59m2.463087944s) [noise=true , address=45.91.116.6:4975 , piece_id=3MBFI2ILY6VP6XT2PMXFVWG7UA2HYOYYNZASSLX3MLQWTEV7EAKA, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.592099669 (430h59m2.463182724s) [noise=true , address=145.79.227.20:4438 , piece_id=OHBU6DZOK7H6MIDZHMXEPL2MDY2OXQSX3VPDIE3RDW2THSMC2IMA, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.593944512 (430h59m2.465027557s) [noise=true , address=145.79.227.16:4910 , piece_id=FDXOMBKKQQPPL3G577PWWCWDJFSGZ3W3Q6OZYANVQ7XSAINJ7MFA, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.597912307 (430h59m2.468995352s) [noise=true , address=145.79.227.29:4943 , piece_id=FF4Q6CY7O4EUFQO5QY7GHSYAAQFHK2WCH7A2BR5B6JGJH4MGHSTA, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.600520093 (430h59m2.471603128s) [noise=true , address=145.79.227.40:4049 , piece_id=3QAECV5EQI3BBHUW7AZXVYWZY6DAOOH6DVP4XAMDXBFJAV34QB5Q, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.602548155 (430h59m2.47363121s) [noise=true , address=45.91.116.4:4939 , piece_id=3BWKNBSDTQ4DBJFYXFY3IPWOJ3JTPPDMRO2DRYA5REJ3BERVMFKQ, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.604654816 (430h59m2.475737861s) [noise=true , address=145.79.227.51:4410 , piece_id=T5XXHOFBLW3A3OKTDY5CTO74VFCY5ZQJRJPACYF3SCLOTF6E25LA, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:04.341061516 (430h59m4.212144561s) [upload=378343]: Next returned nil segment Dec 23 14:28:04.341149894 (430h59m4.212232949s) [upload=378343]: Waiting for error group managing segments... Dec 23 14:28:04.631523518 (430h59m4.502606563s) [noise=true , address=145.79.227.16:4910 , piece_id=FDXOMBKKQQPPL3G577PWWCWDJFSGZ3W3Q6OZYANVQ7XSAINJ7MFA, seg_pos={0 0}, upload=378343]: Done uploading piece. err: <nil> Dec 23 14:28:04.649224493 (430h59m4.520307538s) [noise=true , address=145.79.227.29:4943 , piece_id=FF4Q6CY7O4EUFQO5QY7GHSYAAQFHK2WCH7A2BR5B6JGJH4MGHSTA, seg_pos={0 0}, upload=378343]: Done uploading piece. err: <nil> Dec 23 14:28:04.655099941 (430h59m4.526182986s) [noise=true , address=145.79.227.38:4978 , piece_id=7M2D3NR342SZVWHS754YCZYVPX2SAKZDUHNV5MIUMG7NQYMJORXA, seg_pos={0 0}, upload=378343]: Done uploading piece. err: <nil> Dec 23 14:28:04.663569098 (430h59m4.534652133s) [noise=true , address=145.79.227.15:4873 , piece_id=RROMZU3GQXWHZ2KAHKJNUXB6JYJW5DOZIMS4RTI2CK4H6GJ4JJPA, seg_pos={0 0}, upload=378343]: Done uploading piece. err: <nil> Dec 23 14:28:04.784039696 (430h59m4.655122741s) [noise=true , address=145.79.227.39:4014 , piece_id=UIYHVARR7QJN7TC45DPH7YMWLOV27EVHFR4HNZXTNGCFCYZ5QWSQ, seg_pos={0 0}, upload=378343]: Done uploading piece. err: <nil> Dec 23 14:28:04.789452723 (430h59m4.660535769s) [noise=true , address=145.79.227.50:4375 , piece_id=2L6LRC33345CFQ776H4C3RBHD27J3WN5BBB7QUEQ43EQED5LQJRA, seg_pos={0 0}, upload=378343]: Done uploading piece. err: <nil> Dec 23 14:28:04.809692453 (430h59m4.680775488s) [noise=true , address=145.79.227.32:4834 , piece_id=FLPDIEXB34RV3VYR4LZYW63M76C3G2352VTJPF6IHDJSKAY2VSIQ, seg_pos={0 0}, upload=378343]: Done uploading piece. err: <nil> Dec 23 14:28:04.814157645 (430h59m4.68524069s) [noise=true , address=145.79.227.20:4438 , piece_id=OHBU6DZOK7H6MIDZHMXEPL2MDY2OXQSX3VPDIE3RDW2THSMC2IMA, seg_pos={0 0}, upload=378343]: Done uploading piece. err: <nil> Dec 23 14:28:04.831851378 (430h59m4.702934423s) [noise=true , address=45.91.116.6:4975 , piece_id=3MBFI2ILY6VP6XT2PMXFVWG7UA2HYOYYNZASSLX3MLQWTEV7EAKA, seg_pos={0 0}, upload=378343]: Done uploading piece. err: <nil> Dec 23 14:28:04.884609870 (430h59m4.755692915s) [noise=true , address=45.91.116.4:4939 , piece_id=3BWKNBSDTQ4DBJFYXFY3IPWOJ3JTPPDMRO2DRYA5REJ3BERVMFKQ, seg_pos={0 0}, upload=378343]: Done uploading piece. err: <nil> Dec 23 14:28:05.131970658 (430h59m5.003053693s) [noise=true , address=77.90.31.14:4664 , piece_id=TGN5OFP66O6EI654N63FVJSCOXFSIS3AURJGAGYWLSBHVMAOFYFQ, seg_pos={0 0}, upload=378343]: Done uploading piece. err: <nil> Dec 23 14:28:05.195398928 (430h59m5.066481983s) [noise=true , address=145.79.227.51:4410 , piece_id=T5XXHOFBLW3A3OKTDY5CTO74VFCY5ZQJRJPACYF3SCLOTF6E25LA, seg_pos={0 0}, upload=378343]: Done uploading piece. err: <nil> Dec 23 14:28:05.218188272 (430h59m5.089271317s) [noise=true , address=145.79.227.11:4731 , piece_id=JDCFIGBBNHW67SQCYI7PKCKWPUQBCGO3BZROXPHNABIGFFMATMOQ, seg_pos={0 0}, upload=378343]: Done uploading piece. err: <nil> Dec 23 14:28:05.218214873 (430h59m5.089297908s) [seg_pos={0 0}, upload=378343]: Segment reached optimal threshold of 13 pieces. Dec 23 14:28:05.218460901 (430h59m5.089543966s) [noise=true , address=145.79.227.37:4565 , piece_id=RIXR6277D5GYIDPGNMMSJAQLFCMM4P6LWALPUSNFW67BWRXQJ4SQ, seg_pos={0 0}, upload=378343]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nvvNczHtQrSEJuJbErnjiAWa4BRDzBrQmBNbB7RrrqN1JWNHT): protocol: write tcp 188.166.225.170:39840->145.79.227.37:4565: use of closed network connection; write tcp 188.166.225.170:39840->145.79.227.37:4565: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39840->145.79.227.37:4565: use of closed network connection Dec 23 14:28:05.218469708 (430h59m5.089552753s) [noise=true , address=145.79.227.37:4565 , piece_id=RIXR6277D5GYIDPGNMMSJAQLFCMM4P6LWALPUSNFW67BWRXQJ4SQ, seg_pos={0 0}, upload=378343]: Long tail cancellation due to optimal threshold Dec 23 14:28:05.218750874 (430h59m5.089833919s) [noise=true , address=145.79.227.40:4049 , piece_id=3QAECV5EQI3BBHUW7AZXVYWZY6DAOOH6DVP4XAMDXBFJAV34QB5Q, seg_pos={0 0}, upload=378343]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF Dec 23 14:28:05.218770391 (430h59m5.089853436s) [noise=true , address=145.79.227.40:4049 , piece_id=3QAECV5EQI3BBHUW7AZXVYWZY6DAOOH6DVP4XAMDXBFJAV34QB5Q, seg_pos={0 0}, upload=378343]: Long tail cancellation due to optimal threshold Dec 23 14:28:05.219134284 (430h59m5.090217349s) [noise=true , address=145.79.227.52:4445 , piece_id=VIE24CGHGMPQSCGUQKOGNIOMJMZHYD4BRZSGEPMUSPPAXPSFOHZQ, seg_pos={0 0}, upload=378343]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12vXrkhLqCofhR4Fmfqb98gxnqWQF6vMV4q3ATbRvPfcqi82WSo): protocol: expected piece hash; context canceled; EOF Dec 23 14:28:05.219146146 (430h59m5.090229201s) [noise=true , address=145.79.227.52:4445 , piece_id=VIE24CGHGMPQSCGUQKOGNIOMJMZHYD4BRZSGEPMUSPPAXPSFOHZQ, seg_pos={0 0}, upload=378343]: Long tail cancellation due to optimal threshold Dec 23 14:28:05.219173128 (430h59m5.090256173s) [seg_pos={0 0}, upload=378343]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 14:28:05.219417493 (430h59m5.090500538s) [upload=378343]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 14:28:05.219425228 (430h59m5.090508293s) [upload=378343]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 14:28:05.522079636 (430h59m5.393162671s) [upload=378343]: Done uploading segments... Dec 23 14:28:05.522100636 (430h59m5.393183681s) [upload=378343]: Upload finished. err: <nil> Duration: 7.929784012s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [........X.....X............X................................] duration: 3.809754836s trace: https://jaeger.dev.storj.io/trace/772f273c480c0b2b FAIL (2025-12-23 14:37:39 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.809782269s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.........X.....X............X...............................] duration: 3.680086192s trace: https://jaeger.dev.storj.io/trace/772f273c480c0b2b FAIL (2025-12-23 14:37:17 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.680113934s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [...............X..........X.XX..X............XX..X..........] duration: 5.812450224s trace: https://jaeger.dev.storj.io/trace/215296e6cb5de778 FAIL (2025-12-23 14:36:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9, Piece ID: TGN5OFP66O6EI654N63FVJSCOXFSIS3AURJGAGYWLSBHVMAOFYFQ) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: 3MBFI2ILY6VP6XT2PMXFVWG7UA2HYOYYNZASSLX3MLQWTEV7EAKA) context deadline exceeded ======LOGS====== Dec 23 14:28:02.333620097 (430h59m2.204703152s) [upload=378343]: Starting upload Dec 23 14:28:02.333737752 (430h59m2.204820797s) [upload=378343]: Done starting upload Dec 23 14:28:02.333838052 (430h59m2.204921107s) [upload=378343]: Uploading segments... Dec 23 14:28:02.334165316 (430h59m2.205248381s) [upload=378343]: Got next segment. Inline: false Dec 23 14:28:02.334217274 (430h59m2.205300319s) [upload=378343]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 14:28:02.334693501 (430h59m2.205776567s) [upload=378343]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 14:28:02.588168613 (430h59m2.459251658s) [seg_pos={0 0}, upload=378343]: Begin upload segment... Dec 23 14:28:02.588224871 (430h59m2.459307916s) [seg_pos={0 0}, upload=378343]: Done begin upload segment. Dec 23 14:28:02.588264035 (430h59m2.45934707s) [noise=true , address=145.79.227.32:4834 , piece_id=FLPDIEXB34RV3VYR4LZYW63M76C3G2352VTJPF6IHDJSKAY2VSIQ, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.588315313 (430h59m2.459398349s) [noise=true , address=145.79.227.11:4731 , piece_id=JDCFIGBBNHW67SQCYI7PKCKWPUQBCGO3BZROXPHNABIGFFMATMOQ, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.589826190 (430h59m2.460909225s) [noise=true , address=145.79.227.37:4565 , piece_id=RIXR6277D5GYIDPGNMMSJAQLFCMM4P6LWALPUSNFW67BWRXQJ4SQ, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.589974662 (430h59m2.461057708s) [noise=true , address=145.79.227.50:4375 , piece_id=2L6LRC33345CFQ776H4C3RBHD27J3WN5BBB7QUEQ43EQED5LQJRA, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.590200293 (430h59m2.461283328s) [noise=true , address=145.79.227.52:4445 , piece_id=VIE24CGHGMPQSCGUQKOGNIOMJMZHYD4BRZSGEPMUSPPAXPSFOHZQ, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.590571109 (430h59m2.461654154s) [noise=true , address=145.79.227.39:4014 , piece_id=UIYHVARR7QJN7TC45DPH7YMWLOV27EVHFR4HNZXTNGCFCYZ5QWSQ, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.590591117 (430h59m2.461674162s) [noise=true , address=77.90.31.14:4664 , piece_id=TGN5OFP66O6EI654N63FVJSCOXFSIS3AURJGAGYWLSBHVMAOFYFQ, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.591120444 (430h59m2.462203489s) [noise=true , address=145.79.227.38:4978 , piece_id=7M2D3NR342SZVWHS754YCZYVPX2SAKZDUHNV5MIUMG7NQYMJORXA, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.591482514 (430h59m2.462565559s) [noise=true , address=145.79.227.15:4873 , piece_id=RROMZU3GQXWHZ2KAHKJNUXB6JYJW5DOZIMS4RTI2CK4H6GJ4JJPA, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.592004899 (430h59m2.463087944s) [noise=true , address=45.91.116.6:4975 , piece_id=3MBFI2ILY6VP6XT2PMXFVWG7UA2HYOYYNZASSLX3MLQWTEV7EAKA, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.592099669 (430h59m2.463182724s) [noise=true , address=145.79.227.20:4438 , piece_id=OHBU6DZOK7H6MIDZHMXEPL2MDY2OXQSX3VPDIE3RDW2THSMC2IMA, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.593944512 (430h59m2.465027557s) [noise=true , address=145.79.227.16:4910 , piece_id=FDXOMBKKQQPPL3G577PWWCWDJFSGZ3W3Q6OZYANVQ7XSAINJ7MFA, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.597912307 (430h59m2.468995352s) [noise=true , address=145.79.227.29:4943 , piece_id=FF4Q6CY7O4EUFQO5QY7GHSYAAQFHK2WCH7A2BR5B6JGJH4MGHSTA, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.600520093 (430h59m2.471603128s) [noise=true , address=145.79.227.40:4049 , piece_id=3QAECV5EQI3BBHUW7AZXVYWZY6DAOOH6DVP4XAMDXBFJAV34QB5Q, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.602548155 (430h59m2.47363121s) [noise=true , address=45.91.116.4:4939 , piece_id=3BWKNBSDTQ4DBJFYXFY3IPWOJ3JTPPDMRO2DRYA5REJ3BERVMFKQ, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.604654816 (430h59m2.475737861s) [noise=true , address=145.79.227.51:4410 , piece_id=T5XXHOFBLW3A3OKTDY5CTO74VFCY5ZQJRJPACYF3SCLOTF6E25LA, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:04.341061516 (430h59m4.212144561s) [upload=378343]: Next returned nil segment Dec 23 14:28:04.341149894 (430h59m4.212232949s) [upload=378343]: Waiting for error group managing segments... Dec 23 14:28:04.631523518 (430h59m4.502606563s) [noise=true , address=145.79.227.16:4910 , piece_id=FDXOMBKKQQPPL3G577PWWCWDJFSGZ3W3Q6OZYANVQ7XSAINJ7MFA, seg_pos={0 0}, upload=378343]: Done uploading piece. err: <nil> Dec 23 14:28:04.649224493 (430h59m4.520307538s) [noise=true , address=145.79.227.29:4943 , piece_id=FF4Q6CY7O4EUFQO5QY7GHSYAAQFHK2WCH7A2BR5B6JGJH4MGHSTA, seg_pos={0 0}, upload=378343]: Done uploading piece. err: <nil> Dec 23 14:28:04.655099941 (430h59m4.526182986s) [noise=true , address=145.79.227.38:4978 , piece_id=7M2D3NR342SZVWHS754YCZYVPX2SAKZDUHNV5MIUMG7NQYMJORXA, seg_pos={0 0}, upload=378343]: Done uploading piece. err: <nil> Dec 23 14:28:04.663569098 (430h59m4.534652133s) [noise=true , address=145.79.227.15:4873 , piece_id=RROMZU3GQXWHZ2KAHKJNUXB6JYJW5DOZIMS4RTI2CK4H6GJ4JJPA, seg_pos={0 0}, upload=378343]: Done uploading piece. err: <nil> Dec 23 14:28:04.784039696 (430h59m4.655122741s) [noise=true , address=145.79.227.39:4014 , piece_id=UIYHVARR7QJN7TC45DPH7YMWLOV27EVHFR4HNZXTNGCFCYZ5QWSQ, seg_pos={0 0}, upload=378343]: Done uploading piece. err: <nil> Dec 23 14:28:04.789452723 (430h59m4.660535769s) [noise=true , address=145.79.227.50:4375 , piece_id=2L6LRC33345CFQ776H4C3RBHD27J3WN5BBB7QUEQ43EQED5LQJRA, seg_pos={0 0}, upload=378343]: Done uploading piece. err: <nil> Dec 23 14:28:04.809692453 (430h59m4.680775488s) [noise=true , address=145.79.227.32:4834 , piece_id=FLPDIEXB34RV3VYR4LZYW63M76C3G2352VTJPF6IHDJSKAY2VSIQ, seg_pos={0 0}, upload=378343]: Done uploading piece. err: <nil> Dec 23 14:28:04.814157645 (430h59m4.68524069s) [noise=true , address=145.79.227.20:4438 , piece_id=OHBU6DZOK7H6MIDZHMXEPL2MDY2OXQSX3VPDIE3RDW2THSMC2IMA, seg_pos={0 0}, upload=378343]: Done uploading piece. err: <nil> Dec 23 14:28:04.831851378 (430h59m4.702934423s) [noise=true , address=45.91.116.6:4975 , piece_id=3MBFI2ILY6VP6XT2PMXFVWG7UA2HYOYYNZASSLX3MLQWTEV7EAKA, seg_pos={0 0}, upload=378343]: Done uploading piece. err: <nil> Dec 23 14:28:04.884609870 (430h59m4.755692915s) [noise=true , address=45.91.116.4:4939 , piece_id=3BWKNBSDTQ4DBJFYXFY3IPWOJ3JTPPDMRO2DRYA5REJ3BERVMFKQ, seg_pos={0 0}, upload=378343]: Done uploading piece. err: <nil> Dec 23 14:28:05.131970658 (430h59m5.003053693s) [noise=true , address=77.90.31.14:4664 , piece_id=TGN5OFP66O6EI654N63FVJSCOXFSIS3AURJGAGYWLSBHVMAOFYFQ, seg_pos={0 0}, upload=378343]: Done uploading piece. err: <nil> Dec 23 14:28:05.195398928 (430h59m5.066481983s) [noise=true , address=145.79.227.51:4410 , piece_id=T5XXHOFBLW3A3OKTDY5CTO74VFCY5ZQJRJPACYF3SCLOTF6E25LA, seg_pos={0 0}, upload=378343]: Done uploading piece. err: <nil> Dec 23 14:28:05.218188272 (430h59m5.089271317s) [noise=true , address=145.79.227.11:4731 , piece_id=JDCFIGBBNHW67SQCYI7PKCKWPUQBCGO3BZROXPHNABIGFFMATMOQ, seg_pos={0 0}, upload=378343]: Done uploading piece. err: <nil> Dec 23 14:28:05.218214873 (430h59m5.089297908s) [seg_pos={0 0}, upload=378343]: Segment reached optimal threshold of 13 pieces. Dec 23 14:28:05.218460901 (430h59m5.089543966s) [noise=true , address=145.79.227.37:4565 , piece_id=RIXR6277D5GYIDPGNMMSJAQLFCMM4P6LWALPUSNFW67BWRXQJ4SQ, seg_pos={0 0}, upload=378343]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nvvNczHtQrSEJuJbErnjiAWa4BRDzBrQmBNbB7RrrqN1JWNHT): protocol: write tcp 188.166.225.170:39840->145.79.227.37:4565: use of closed network connection; write tcp 188.166.225.170:39840->145.79.227.37:4565: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39840->145.79.227.37:4565: use of closed network connection Dec 23 14:28:05.218469708 (430h59m5.089552753s) [noise=true , address=145.79.227.37:4565 , piece_id=RIXR6277D5GYIDPGNMMSJAQLFCMM4P6LWALPUSNFW67BWRXQJ4SQ, seg_pos={0 0}, upload=378343]: Long tail cancellation due to optimal threshold Dec 23 14:28:05.218750874 (430h59m5.089833919s) [noise=true , address=145.79.227.40:4049 , piece_id=3QAECV5EQI3BBHUW7AZXVYWZY6DAOOH6DVP4XAMDXBFJAV34QB5Q, seg_pos={0 0}, upload=378343]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF Dec 23 14:28:05.218770391 (430h59m5.089853436s) [noise=true , address=145.79.227.40:4049 , piece_id=3QAECV5EQI3BBHUW7AZXVYWZY6DAOOH6DVP4XAMDXBFJAV34QB5Q, seg_pos={0 0}, upload=378343]: Long tail cancellation due to optimal threshold Dec 23 14:28:05.219134284 (430h59m5.090217349s) [noise=true , address=145.79.227.52:4445 , piece_id=VIE24CGHGMPQSCGUQKOGNIOMJMZHYD4BRZSGEPMUSPPAXPSFOHZQ, seg_pos={0 0}, upload=378343]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12vXrkhLqCofhR4Fmfqb98gxnqWQF6vMV4q3ATbRvPfcqi82WSo): protocol: expected piece hash; context canceled; EOF Dec 23 14:28:05.219146146 (430h59m5.090229201s) [noise=true , address=145.79.227.52:4445 , piece_id=VIE24CGHGMPQSCGUQKOGNIOMJMZHYD4BRZSGEPMUSPPAXPSFOHZQ, seg_pos={0 0}, upload=378343]: Long tail cancellation due to optimal threshold Dec 23 14:28:05.219173128 (430h59m5.090256173s) [seg_pos={0 0}, upload=378343]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 14:28:05.219417493 (430h59m5.090500538s) [upload=378343]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 14:28:05.219425228 (430h59m5.090508293s) [upload=378343]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 14:28:05.522079636 (430h59m5.393162671s) [upload=378343]: Done uploading segments... Dec 23 14:28:05.522100636 (430h59m5.393183681s) [upload=378343]: Upload finished. err: <nil> Duration: 5.812476633s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [..........X.....X............X..............................] duration: 4.045616459s trace: https://jaeger.dev.storj.io/trace/772f273c480c0b2b FAIL (2025-12-23 14:36:10 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.045644754s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [................X..........X.XX..X............XX..X.........] duration: 6.224643906s trace: https://jaeger.dev.storj.io/trace/215296e6cb5de778 FAIL (2025-12-23 14:35:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9, Piece ID: TGN5OFP66O6EI654N63FVJSCOXFSIS3AURJGAGYWLSBHVMAOFYFQ) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: 3MBFI2ILY6VP6XT2PMXFVWG7UA2HYOYYNZASSLX3MLQWTEV7EAKA) context deadline exceeded ======LOGS====== Dec 23 14:28:02.333620097 (430h59m2.204703152s) [upload=378343]: Starting upload Dec 23 14:28:02.333737752 (430h59m2.204820797s) [upload=378343]: Done starting upload Dec 23 14:28:02.333838052 (430h59m2.204921107s) [upload=378343]: Uploading segments... Dec 23 14:28:02.334165316 (430h59m2.205248381s) [upload=378343]: Got next segment. Inline: false Dec 23 14:28:02.334217274 (430h59m2.205300319s) [upload=378343]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 14:28:02.334693501 (430h59m2.205776567s) [upload=378343]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 14:28:02.588168613 (430h59m2.459251658s) [seg_pos={0 0}, upload=378343]: Begin upload segment... Dec 23 14:28:02.588224871 (430h59m2.459307916s) [seg_pos={0 0}, upload=378343]: Done begin upload segment. Dec 23 14:28:02.588264035 (430h59m2.45934707s) [noise=true , address=145.79.227.32:4834 , piece_id=FLPDIEXB34RV3VYR4LZYW63M76C3G2352VTJPF6IHDJSKAY2VSIQ, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.588315313 (430h59m2.459398349s) [noise=true , address=145.79.227.11:4731 , piece_id=JDCFIGBBNHW67SQCYI7PKCKWPUQBCGO3BZROXPHNABIGFFMATMOQ, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.589826190 (430h59m2.460909225s) [noise=true , address=145.79.227.37:4565 , piece_id=RIXR6277D5GYIDPGNMMSJAQLFCMM4P6LWALPUSNFW67BWRXQJ4SQ, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.589974662 (430h59m2.461057708s) [noise=true , address=145.79.227.50:4375 , piece_id=2L6LRC33345CFQ776H4C3RBHD27J3WN5BBB7QUEQ43EQED5LQJRA, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.590200293 (430h59m2.461283328s) [noise=true , address=145.79.227.52:4445 , piece_id=VIE24CGHGMPQSCGUQKOGNIOMJMZHYD4BRZSGEPMUSPPAXPSFOHZQ, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.590571109 (430h59m2.461654154s) [noise=true , address=145.79.227.39:4014 , piece_id=UIYHVARR7QJN7TC45DPH7YMWLOV27EVHFR4HNZXTNGCFCYZ5QWSQ, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.590591117 (430h59m2.461674162s) [noise=true , address=77.90.31.14:4664 , piece_id=TGN5OFP66O6EI654N63FVJSCOXFSIS3AURJGAGYWLSBHVMAOFYFQ, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.591120444 (430h59m2.462203489s) [noise=true , address=145.79.227.38:4978 , piece_id=7M2D3NR342SZVWHS754YCZYVPX2SAKZDUHNV5MIUMG7NQYMJORXA, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.591482514 (430h59m2.462565559s) [noise=true , address=145.79.227.15:4873 , piece_id=RROMZU3GQXWHZ2KAHKJNUXB6JYJW5DOZIMS4RTI2CK4H6GJ4JJPA, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.592004899 (430h59m2.463087944s) [noise=true , address=45.91.116.6:4975 , piece_id=3MBFI2ILY6VP6XT2PMXFVWG7UA2HYOYYNZASSLX3MLQWTEV7EAKA, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.592099669 (430h59m2.463182724s) [noise=true , address=145.79.227.20:4438 , piece_id=OHBU6DZOK7H6MIDZHMXEPL2MDY2OXQSX3VPDIE3RDW2THSMC2IMA, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.593944512 (430h59m2.465027557s) [noise=true , address=145.79.227.16:4910 , piece_id=FDXOMBKKQQPPL3G577PWWCWDJFSGZ3W3Q6OZYANVQ7XSAINJ7MFA, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.597912307 (430h59m2.468995352s) [noise=true , address=145.79.227.29:4943 , piece_id=FF4Q6CY7O4EUFQO5QY7GHSYAAQFHK2WCH7A2BR5B6JGJH4MGHSTA, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.600520093 (430h59m2.471603128s) [noise=true , address=145.79.227.40:4049 , piece_id=3QAECV5EQI3BBHUW7AZXVYWZY6DAOOH6DVP4XAMDXBFJAV34QB5Q, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.602548155 (430h59m2.47363121s) [noise=true , address=45.91.116.4:4939 , piece_id=3BWKNBSDTQ4DBJFYXFY3IPWOJ3JTPPDMRO2DRYA5REJ3BERVMFKQ, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.604654816 (430h59m2.475737861s) [noise=true , address=145.79.227.51:4410 , piece_id=T5XXHOFBLW3A3OKTDY5CTO74VFCY5ZQJRJPACYF3SCLOTF6E25LA, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:04.341061516 (430h59m4.212144561s) [upload=378343]: Next returned nil segment Dec 23 14:28:04.341149894 (430h59m4.212232949s) [upload=378343]: Waiting for error group managing segments... Dec 23 14:28:04.631523518 (430h59m4.502606563s) [noise=true , address=145.79.227.16:4910 , piece_id=FDXOMBKKQQPPL3G577PWWCWDJFSGZ3W3Q6OZYANVQ7XSAINJ7MFA, seg_pos={0 0}, upload=378343]: Done uploading piece. err: <nil> Dec 23 14:28:04.649224493 (430h59m4.520307538s) [noise=true , address=145.79.227.29:4943 , piece_id=FF4Q6CY7O4EUFQO5QY7GHSYAAQFHK2WCH7A2BR5B6JGJH4MGHSTA, seg_pos={0 0}, upload=378343]: Done uploading piece. err: <nil> Dec 23 14:28:04.655099941 (430h59m4.526182986s) [noise=true , address=145.79.227.38:4978 , piece_id=7M2D3NR342SZVWHS754YCZYVPX2SAKZDUHNV5MIUMG7NQYMJORXA, seg_pos={0 0}, upload=378343]: Done uploading piece. err: <nil> Dec 23 14:28:04.663569098 (430h59m4.534652133s) [noise=true , address=145.79.227.15:4873 , piece_id=RROMZU3GQXWHZ2KAHKJNUXB6JYJW5DOZIMS4RTI2CK4H6GJ4JJPA, seg_pos={0 0}, upload=378343]: Done uploading piece. err: <nil> Dec 23 14:28:04.784039696 (430h59m4.655122741s) [noise=true , address=145.79.227.39:4014 , piece_id=UIYHVARR7QJN7TC45DPH7YMWLOV27EVHFR4HNZXTNGCFCYZ5QWSQ, seg_pos={0 0}, upload=378343]: Done uploading piece. err: <nil> Dec 23 14:28:04.789452723 (430h59m4.660535769s) [noise=true , address=145.79.227.50:4375 , piece_id=2L6LRC33345CFQ776H4C3RBHD27J3WN5BBB7QUEQ43EQED5LQJRA, seg_pos={0 0}, upload=378343]: Done uploading piece. err: <nil> Dec 23 14:28:04.809692453 (430h59m4.680775488s) [noise=true , address=145.79.227.32:4834 , piece_id=FLPDIEXB34RV3VYR4LZYW63M76C3G2352VTJPF6IHDJSKAY2VSIQ, seg_pos={0 0}, upload=378343]: Done uploading piece. err: <nil> Dec 23 14:28:04.814157645 (430h59m4.68524069s) [noise=true , address=145.79.227.20:4438 , piece_id=OHBU6DZOK7H6MIDZHMXEPL2MDY2OXQSX3VPDIE3RDW2THSMC2IMA, seg_pos={0 0}, upload=378343]: Done uploading piece. err: <nil> Dec 23 14:28:04.831851378 (430h59m4.702934423s) [noise=true , address=45.91.116.6:4975 , piece_id=3MBFI2ILY6VP6XT2PMXFVWG7UA2HYOYYNZASSLX3MLQWTEV7EAKA, seg_pos={0 0}, upload=378343]: Done uploading piece. err: <nil> Dec 23 14:28:04.884609870 (430h59m4.755692915s) [noise=true , address=45.91.116.4:4939 , piece_id=3BWKNBSDTQ4DBJFYXFY3IPWOJ3JTPPDMRO2DRYA5REJ3BERVMFKQ, seg_pos={0 0}, upload=378343]: Done uploading piece. err: <nil> Dec 23 14:28:05.131970658 (430h59m5.003053693s) [noise=true , address=77.90.31.14:4664 , piece_id=TGN5OFP66O6EI654N63FVJSCOXFSIS3AURJGAGYWLSBHVMAOFYFQ, seg_pos={0 0}, upload=378343]: Done uploading piece. err: <nil> Dec 23 14:28:05.195398928 (430h59m5.066481983s) [noise=true , address=145.79.227.51:4410 , piece_id=T5XXHOFBLW3A3OKTDY5CTO74VFCY5ZQJRJPACYF3SCLOTF6E25LA, seg_pos={0 0}, upload=378343]: Done uploading piece. err: <nil> Dec 23 14:28:05.218188272 (430h59m5.089271317s) [noise=true , address=145.79.227.11:4731 , piece_id=JDCFIGBBNHW67SQCYI7PKCKWPUQBCGO3BZROXPHNABIGFFMATMOQ, seg_pos={0 0}, upload=378343]: Done uploading piece. err: <nil> Dec 23 14:28:05.218214873 (430h59m5.089297908s) [seg_pos={0 0}, upload=378343]: Segment reached optimal threshold of 13 pieces. Dec 23 14:28:05.218460901 (430h59m5.089543966s) [noise=true , address=145.79.227.37:4565 , piece_id=RIXR6277D5GYIDPGNMMSJAQLFCMM4P6LWALPUSNFW67BWRXQJ4SQ, seg_pos={0 0}, upload=378343]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nvvNczHtQrSEJuJbErnjiAWa4BRDzBrQmBNbB7RrrqN1JWNHT): protocol: write tcp 188.166.225.170:39840->145.79.227.37:4565: use of closed network connection; write tcp 188.166.225.170:39840->145.79.227.37:4565: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39840->145.79.227.37:4565: use of closed network connection Dec 23 14:28:05.218469708 (430h59m5.089552753s) [noise=true , address=145.79.227.37:4565 , piece_id=RIXR6277D5GYIDPGNMMSJAQLFCMM4P6LWALPUSNFW67BWRXQJ4SQ, seg_pos={0 0}, upload=378343]: Long tail cancellation due to optimal threshold Dec 23 14:28:05.218750874 (430h59m5.089833919s) [noise=true , address=145.79.227.40:4049 , piece_id=3QAECV5EQI3BBHUW7AZXVYWZY6DAOOH6DVP4XAMDXBFJAV34QB5Q, seg_pos={0 0}, upload=378343]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF Dec 23 14:28:05.218770391 (430h59m5.089853436s) [noise=true , address=145.79.227.40:4049 , piece_id=3QAECV5EQI3BBHUW7AZXVYWZY6DAOOH6DVP4XAMDXBFJAV34QB5Q, seg_pos={0 0}, upload=378343]: Long tail cancellation due to optimal threshold Dec 23 14:28:05.219134284 (430h59m5.090217349s) [noise=true , address=145.79.227.52:4445 , piece_id=VIE24CGHGMPQSCGUQKOGNIOMJMZHYD4BRZSGEPMUSPPAXPSFOHZQ, seg_pos={0 0}, upload=378343]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12vXrkhLqCofhR4Fmfqb98gxnqWQF6vMV4q3ATbRvPfcqi82WSo): protocol: expected piece hash; context canceled; EOF Dec 23 14:28:05.219146146 (430h59m5.090229201s) [noise=true , address=145.79.227.52:4445 , piece_id=VIE24CGHGMPQSCGUQKOGNIOMJMZHYD4BRZSGEPMUSPPAXPSFOHZQ, seg_pos={0 0}, upload=378343]: Long tail cancellation due to optimal threshold Dec 23 14:28:05.219173128 (430h59m5.090256173s) [seg_pos={0 0}, upload=378343]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 14:28:05.219417493 (430h59m5.090500538s) [upload=378343]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 14:28:05.219425228 (430h59m5.090508293s) [upload=378343]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 14:28:05.522079636 (430h59m5.393162671s) [upload=378343]: Done uploading segments... Dec 23 14:28:05.522100636 (430h59m5.393183681s) [upload=378343]: Upload finished. err: <nil> Duration: 6.224674205s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [...........X.....X............X.............................] duration: 3.779988255s trace: https://jaeger.dev.storj.io/trace/772f273c480c0b2b FAIL (2025-12-23 14:35:23 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.780006149s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.................X..........X.XX..X............XX..X........] duration: 12.521709015s trace: https://jaeger.dev.storj.io/trace/215296e6cb5de778 FAIL (2025-12-23 14:34:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9, Piece ID: TGN5OFP66O6EI654N63FVJSCOXFSIS3AURJGAGYWLSBHVMAOFYFQ) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: 3MBFI2ILY6VP6XT2PMXFVWG7UA2HYOYYNZASSLX3MLQWTEV7EAKA) context deadline exceeded ======LOGS====== Dec 23 14:28:02.333620097 (430h59m2.204703152s) [upload=378343]: Starting upload Dec 23 14:28:02.333737752 (430h59m2.204820797s) [upload=378343]: Done starting upload Dec 23 14:28:02.333838052 (430h59m2.204921107s) [upload=378343]: Uploading segments... Dec 23 14:28:02.334165316 (430h59m2.205248381s) [upload=378343]: Got next segment. Inline: false Dec 23 14:28:02.334217274 (430h59m2.205300319s) [upload=378343]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 14:28:02.334693501 (430h59m2.205776567s) [upload=378343]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 14:28:02.588168613 (430h59m2.459251658s) [seg_pos={0 0}, upload=378343]: Begin upload segment... Dec 23 14:28:02.588224871 (430h59m2.459307916s) [seg_pos={0 0}, upload=378343]: Done begin upload segment. Dec 23 14:28:02.588264035 (430h59m2.45934707s) [noise=true , address=145.79.227.32:4834 , piece_id=FLPDIEXB34RV3VYR4LZYW63M76C3G2352VTJPF6IHDJSKAY2VSIQ, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.588315313 (430h59m2.459398349s) [noise=true , address=145.79.227.11:4731 , piece_id=JDCFIGBBNHW67SQCYI7PKCKWPUQBCGO3BZROXPHNABIGFFMATMOQ, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.589826190 (430h59m2.460909225s) [noise=true , address=145.79.227.37:4565 , piece_id=RIXR6277D5GYIDPGNMMSJAQLFCMM4P6LWALPUSNFW67BWRXQJ4SQ, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.589974662 (430h59m2.461057708s) [noise=true , address=145.79.227.50:4375 , piece_id=2L6LRC33345CFQ776H4C3RBHD27J3WN5BBB7QUEQ43EQED5LQJRA, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.590200293 (430h59m2.461283328s) [noise=true , address=145.79.227.52:4445 , piece_id=VIE24CGHGMPQSCGUQKOGNIOMJMZHYD4BRZSGEPMUSPPAXPSFOHZQ, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.590571109 (430h59m2.461654154s) [noise=true , address=145.79.227.39:4014 , piece_id=UIYHVARR7QJN7TC45DPH7YMWLOV27EVHFR4HNZXTNGCFCYZ5QWSQ, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.590591117 (430h59m2.461674162s) [noise=true , address=77.90.31.14:4664 , piece_id=TGN5OFP66O6EI654N63FVJSCOXFSIS3AURJGAGYWLSBHVMAOFYFQ, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.591120444 (430h59m2.462203489s) [noise=true , address=145.79.227.38:4978 , piece_id=7M2D3NR342SZVWHS754YCZYVPX2SAKZDUHNV5MIUMG7NQYMJORXA, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.591482514 (430h59m2.462565559s) [noise=true , address=145.79.227.15:4873 , piece_id=RROMZU3GQXWHZ2KAHKJNUXB6JYJW5DOZIMS4RTI2CK4H6GJ4JJPA, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.592004899 (430h59m2.463087944s) [noise=true , address=45.91.116.6:4975 , piece_id=3MBFI2ILY6VP6XT2PMXFVWG7UA2HYOYYNZASSLX3MLQWTEV7EAKA, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.592099669 (430h59m2.463182724s) [noise=true , address=145.79.227.20:4438 , piece_id=OHBU6DZOK7H6MIDZHMXEPL2MDY2OXQSX3VPDIE3RDW2THSMC2IMA, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.593944512 (430h59m2.465027557s) [noise=true , address=145.79.227.16:4910 , piece_id=FDXOMBKKQQPPL3G577PWWCWDJFSGZ3W3Q6OZYANVQ7XSAINJ7MFA, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.597912307 (430h59m2.468995352s) [noise=true , address=145.79.227.29:4943 , piece_id=FF4Q6CY7O4EUFQO5QY7GHSYAAQFHK2WCH7A2BR5B6JGJH4MGHSTA, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.600520093 (430h59m2.471603128s) [noise=true , address=145.79.227.40:4049 , piece_id=3QAECV5EQI3BBHUW7AZXVYWZY6DAOOH6DVP4XAMDXBFJAV34QB5Q, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.602548155 (430h59m2.47363121s) [noise=true , address=45.91.116.4:4939 , piece_id=3BWKNBSDTQ4DBJFYXFY3IPWOJ3JTPPDMRO2DRYA5REJ3BERVMFKQ, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.604654816 (430h59m2.475737861s) [noise=true , address=145.79.227.51:4410 , piece_id=T5XXHOFBLW3A3OKTDY5CTO74VFCY5ZQJRJPACYF3SCLOTF6E25LA, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:04.341061516 (430h59m4.212144561s) [upload=378343]: Next returned nil segment Dec 23 14:28:04.341149894 (430h59m4.212232949s) [upload=378343]: Waiting for error group managing segments... Dec 23 14:28:04.631523518 (430h59m4.502606563s) [noise=true , address=145.79.227.16:4910 , piece_id=FDXOMBKKQQPPL3G577PWWCWDJFSGZ3W3Q6OZYANVQ7XSAINJ7MFA, seg_pos={0 0}, upload=378343]: Done uploading piece. err: <nil> Dec 23 14:28:04.649224493 (430h59m4.520307538s) [noise=true , address=145.79.227.29:4943 , piece_id=FF4Q6CY7O4EUFQO5QY7GHSYAAQFHK2WCH7A2BR5B6JGJH4MGHSTA, seg_pos={0 0}, upload=378343]: Done uploading piece. err: <nil> Dec 23 14:28:04.655099941 (430h59m4.526182986s) [noise=true , address=145.79.227.38:4978 , piece_id=7M2D3NR342SZVWHS754YCZYVPX2SAKZDUHNV5MIUMG7NQYMJORXA, seg_pos={0 0}, upload=378343]: Done uploading piece. err: <nil> Dec 23 14:28:04.663569098 (430h59m4.534652133s) [noise=true , address=145.79.227.15:4873 , piece_id=RROMZU3GQXWHZ2KAHKJNUXB6JYJW5DOZIMS4RTI2CK4H6GJ4JJPA, seg_pos={0 0}, upload=378343]: Done uploading piece. err: <nil> Dec 23 14:28:04.784039696 (430h59m4.655122741s) [noise=true , address=145.79.227.39:4014 , piece_id=UIYHVARR7QJN7TC45DPH7YMWLOV27EVHFR4HNZXTNGCFCYZ5QWSQ, seg_pos={0 0}, upload=378343]: Done uploading piece. err: <nil> Dec 23 14:28:04.789452723 (430h59m4.660535769s) [noise=true , address=145.79.227.50:4375 , piece_id=2L6LRC33345CFQ776H4C3RBHD27J3WN5BBB7QUEQ43EQED5LQJRA, seg_pos={0 0}, upload=378343]: Done uploading piece. err: <nil> Dec 23 14:28:04.809692453 (430h59m4.680775488s) [noise=true , address=145.79.227.32:4834 , piece_id=FLPDIEXB34RV3VYR4LZYW63M76C3G2352VTJPF6IHDJSKAY2VSIQ, seg_pos={0 0}, upload=378343]: Done uploading piece. err: <nil> Dec 23 14:28:04.814157645 (430h59m4.68524069s) [noise=true , address=145.79.227.20:4438 , piece_id=OHBU6DZOK7H6MIDZHMXEPL2MDY2OXQSX3VPDIE3RDW2THSMC2IMA, seg_pos={0 0}, upload=378343]: Done uploading piece. err: <nil> Dec 23 14:28:04.831851378 (430h59m4.702934423s) [noise=true , address=45.91.116.6:4975 , piece_id=3MBFI2ILY6VP6XT2PMXFVWG7UA2HYOYYNZASSLX3MLQWTEV7EAKA, seg_pos={0 0}, upload=378343]: Done uploading piece. err: <nil> Dec 23 14:28:04.884609870 (430h59m4.755692915s) [noise=true , address=45.91.116.4:4939 , piece_id=3BWKNBSDTQ4DBJFYXFY3IPWOJ3JTPPDMRO2DRYA5REJ3BERVMFKQ, seg_pos={0 0}, upload=378343]: Done uploading piece. err: <nil> Dec 23 14:28:05.131970658 (430h59m5.003053693s) [noise=true , address=77.90.31.14:4664 , piece_id=TGN5OFP66O6EI654N63FVJSCOXFSIS3AURJGAGYWLSBHVMAOFYFQ, seg_pos={0 0}, upload=378343]: Done uploading piece. err: <nil> Dec 23 14:28:05.195398928 (430h59m5.066481983s) [noise=true , address=145.79.227.51:4410 , piece_id=T5XXHOFBLW3A3OKTDY5CTO74VFCY5ZQJRJPACYF3SCLOTF6E25LA, seg_pos={0 0}, upload=378343]: Done uploading piece. err: <nil> Dec 23 14:28:05.218188272 (430h59m5.089271317s) [noise=true , address=145.79.227.11:4731 , piece_id=JDCFIGBBNHW67SQCYI7PKCKWPUQBCGO3BZROXPHNABIGFFMATMOQ, seg_pos={0 0}, upload=378343]: Done uploading piece. err: <nil> Dec 23 14:28:05.218214873 (430h59m5.089297908s) [seg_pos={0 0}, upload=378343]: Segment reached optimal threshold of 13 pieces. Dec 23 14:28:05.218460901 (430h59m5.089543966s) [noise=true , address=145.79.227.37:4565 , piece_id=RIXR6277D5GYIDPGNMMSJAQLFCMM4P6LWALPUSNFW67BWRXQJ4SQ, seg_pos={0 0}, upload=378343]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nvvNczHtQrSEJuJbErnjiAWa4BRDzBrQmBNbB7RrrqN1JWNHT): protocol: write tcp 188.166.225.170:39840->145.79.227.37:4565: use of closed network connection; write tcp 188.166.225.170:39840->145.79.227.37:4565: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39840->145.79.227.37:4565: use of closed network connection Dec 23 14:28:05.218469708 (430h59m5.089552753s) [noise=true , address=145.79.227.37:4565 , piece_id=RIXR6277D5GYIDPGNMMSJAQLFCMM4P6LWALPUSNFW67BWRXQJ4SQ, seg_pos={0 0}, upload=378343]: Long tail cancellation due to optimal threshold Dec 23 14:28:05.218750874 (430h59m5.089833919s) [noise=true , address=145.79.227.40:4049 , piece_id=3QAECV5EQI3BBHUW7AZXVYWZY6DAOOH6DVP4XAMDXBFJAV34QB5Q, seg_pos={0 0}, upload=378343]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF Dec 23 14:28:05.218770391 (430h59m5.089853436s) [noise=true , address=145.79.227.40:4049 , piece_id=3QAECV5EQI3BBHUW7AZXVYWZY6DAOOH6DVP4XAMDXBFJAV34QB5Q, seg_pos={0 0}, upload=378343]: Long tail cancellation due to optimal threshold Dec 23 14:28:05.219134284 (430h59m5.090217349s) [noise=true , address=145.79.227.52:4445 , piece_id=VIE24CGHGMPQSCGUQKOGNIOMJMZHYD4BRZSGEPMUSPPAXPSFOHZQ, seg_pos={0 0}, upload=378343]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12vXrkhLqCofhR4Fmfqb98gxnqWQF6vMV4q3ATbRvPfcqi82WSo): protocol: expected piece hash; context canceled; EOF Dec 23 14:28:05.219146146 (430h59m5.090229201s) [noise=true , address=145.79.227.52:4445 , piece_id=VIE24CGHGMPQSCGUQKOGNIOMJMZHYD4BRZSGEPMUSPPAXPSFOHZQ, seg_pos={0 0}, upload=378343]: Long tail cancellation due to optimal threshold Dec 23 14:28:05.219173128 (430h59m5.090256173s) [seg_pos={0 0}, upload=378343]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 14:28:05.219417493 (430h59m5.090500538s) [upload=378343]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 14:28:05.219425228 (430h59m5.090508293s) [upload=378343]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 14:28:05.522079636 (430h59m5.393162671s) [upload=378343]: Done uploading segments... Dec 23 14:28:05.522100636 (430h59m5.393183681s) [upload=378343]: Upload finished. err: <nil> Duration: 12.521738851s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [............X.....X............X............................] duration: 3.702017464s trace: https://jaeger.dev.storj.io/trace/772f273c480c0b2b FAIL (2025-12-23 14:34:22 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.702042461s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [..................X..........X.XX..X............XX..X.......] duration: 6.805265514s trace: https://jaeger.dev.storj.io/trace/215296e6cb5de778 FAIL (2025-12-23 14:33:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9, Piece ID: TGN5OFP66O6EI654N63FVJSCOXFSIS3AURJGAGYWLSBHVMAOFYFQ) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: 3MBFI2ILY6VP6XT2PMXFVWG7UA2HYOYYNZASSLX3MLQWTEV7EAKA) context deadline exceeded ======LOGS====== Dec 23 14:28:02.333620097 (430h59m2.204703152s) [upload=378343]: Starting upload Dec 23 14:28:02.333737752 (430h59m2.204820797s) [upload=378343]: Done starting upload Dec 23 14:28:02.333838052 (430h59m2.204921107s) [upload=378343]: Uploading segments... Dec 23 14:28:02.334165316 (430h59m2.205248381s) [upload=378343]: Got next segment. Inline: false Dec 23 14:28:02.334217274 (430h59m2.205300319s) [upload=378343]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 14:28:02.334693501 (430h59m2.205776567s) [upload=378343]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 14:28:02.588168613 (430h59m2.459251658s) [seg_pos={0 0}, upload=378343]: Begin upload segment... Dec 23 14:28:02.588224871 (430h59m2.459307916s) [seg_pos={0 0}, upload=378343]: Done begin upload segment. Dec 23 14:28:02.588264035 (430h59m2.45934707s) [noise=true , address=145.79.227.32:4834 , piece_id=FLPDIEXB34RV3VYR4LZYW63M76C3G2352VTJPF6IHDJSKAY2VSIQ, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.588315313 (430h59m2.459398349s) [noise=true , address=145.79.227.11:4731 , piece_id=JDCFIGBBNHW67SQCYI7PKCKWPUQBCGO3BZROXPHNABIGFFMATMOQ, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.589826190 (430h59m2.460909225s) [noise=true , address=145.79.227.37:4565 , piece_id=RIXR6277D5GYIDPGNMMSJAQLFCMM4P6LWALPUSNFW67BWRXQJ4SQ, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.589974662 (430h59m2.461057708s) [noise=true , address=145.79.227.50:4375 , piece_id=2L6LRC33345CFQ776H4C3RBHD27J3WN5BBB7QUEQ43EQED5LQJRA, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.590200293 (430h59m2.461283328s) [noise=true , address=145.79.227.52:4445 , piece_id=VIE24CGHGMPQSCGUQKOGNIOMJMZHYD4BRZSGEPMUSPPAXPSFOHZQ, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.590571109 (430h59m2.461654154s) [noise=true , address=145.79.227.39:4014 , piece_id=UIYHVARR7QJN7TC45DPH7YMWLOV27EVHFR4HNZXTNGCFCYZ5QWSQ, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.590591117 (430h59m2.461674162s) [noise=true , address=77.90.31.14:4664 , piece_id=TGN5OFP66O6EI654N63FVJSCOXFSIS3AURJGAGYWLSBHVMAOFYFQ, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.591120444 (430h59m2.462203489s) [noise=true , address=145.79.227.38:4978 , piece_id=7M2D3NR342SZVWHS754YCZYVPX2SAKZDUHNV5MIUMG7NQYMJORXA, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.591482514 (430h59m2.462565559s) [noise=true , address=145.79.227.15:4873 , piece_id=RROMZU3GQXWHZ2KAHKJNUXB6JYJW5DOZIMS4RTI2CK4H6GJ4JJPA, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.592004899 (430h59m2.463087944s) [noise=true , address=45.91.116.6:4975 , piece_id=3MBFI2ILY6VP6XT2PMXFVWG7UA2HYOYYNZASSLX3MLQWTEV7EAKA, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.592099669 (430h59m2.463182724s) [noise=true , address=145.79.227.20:4438 , piece_id=OHBU6DZOK7H6MIDZHMXEPL2MDY2OXQSX3VPDIE3RDW2THSMC2IMA, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.593944512 (430h59m2.465027557s) [noise=true , address=145.79.227.16:4910 , piece_id=FDXOMBKKQQPPL3G577PWWCWDJFSGZ3W3Q6OZYANVQ7XSAINJ7MFA, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.597912307 (430h59m2.468995352s) [noise=true , address=145.79.227.29:4943 , piece_id=FF4Q6CY7O4EUFQO5QY7GHSYAAQFHK2WCH7A2BR5B6JGJH4MGHSTA, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.600520093 (430h59m2.471603128s) [noise=true , address=145.79.227.40:4049 , piece_id=3QAECV5EQI3BBHUW7AZXVYWZY6DAOOH6DVP4XAMDXBFJAV34QB5Q, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.602548155 (430h59m2.47363121s) [noise=true , address=45.91.116.4:4939 , piece_id=3BWKNBSDTQ4DBJFYXFY3IPWOJ3JTPPDMRO2DRYA5REJ3BERVMFKQ, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.604654816 (430h59m2.475737861s) [noise=true , address=145.79.227.51:4410 , piece_id=T5XXHOFBLW3A3OKTDY5CTO74VFCY5ZQJRJPACYF3SCLOTF6E25LA, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:04.341061516 (430h59m4.212144561s) [upload=378343]: Next returned nil segment Dec 23 14:28:04.341149894 (430h59m4.212232949s) [upload=378343]: Waiting for error group managing segments... Dec 23 14:28:04.631523518 (430h59m4.502606563s) [noise=true , address=145.79.227.16:4910 , piece_id=FDXOMBKKQQPPL3G577PWWCWDJFSGZ3W3Q6OZYANVQ7XSAINJ7MFA, seg_pos={0 0}, upload=378343]: Done uploading piece. err: <nil> Dec 23 14:28:04.649224493 (430h59m4.520307538s) [noise=true , address=145.79.227.29:4943 , piece_id=FF4Q6CY7O4EUFQO5QY7GHSYAAQFHK2WCH7A2BR5B6JGJH4MGHSTA, seg_pos={0 0}, upload=378343]: Done uploading piece. err: <nil> Dec 23 14:28:04.655099941 (430h59m4.526182986s) [noise=true , address=145.79.227.38:4978 , piece_id=7M2D3NR342SZVWHS754YCZYVPX2SAKZDUHNV5MIUMG7NQYMJORXA, seg_pos={0 0}, upload=378343]: Done uploading piece. err: <nil> Dec 23 14:28:04.663569098 (430h59m4.534652133s) [noise=true , address=145.79.227.15:4873 , piece_id=RROMZU3GQXWHZ2KAHKJNUXB6JYJW5DOZIMS4RTI2CK4H6GJ4JJPA, seg_pos={0 0}, upload=378343]: Done uploading piece. err: <nil> Dec 23 14:28:04.784039696 (430h59m4.655122741s) [noise=true , address=145.79.227.39:4014 , piece_id=UIYHVARR7QJN7TC45DPH7YMWLOV27EVHFR4HNZXTNGCFCYZ5QWSQ, seg_pos={0 0}, upload=378343]: Done uploading piece. err: <nil> Dec 23 14:28:04.789452723 (430h59m4.660535769s) [noise=true , address=145.79.227.50:4375 , piece_id=2L6LRC33345CFQ776H4C3RBHD27J3WN5BBB7QUEQ43EQED5LQJRA, seg_pos={0 0}, upload=378343]: Done uploading piece. err: <nil> Dec 23 14:28:04.809692453 (430h59m4.680775488s) [noise=true , address=145.79.227.32:4834 , piece_id=FLPDIEXB34RV3VYR4LZYW63M76C3G2352VTJPF6IHDJSKAY2VSIQ, seg_pos={0 0}, upload=378343]: Done uploading piece. err: <nil> Dec 23 14:28:04.814157645 (430h59m4.68524069s) [noise=true , address=145.79.227.20:4438 , piece_id=OHBU6DZOK7H6MIDZHMXEPL2MDY2OXQSX3VPDIE3RDW2THSMC2IMA, seg_pos={0 0}, upload=378343]: Done uploading piece. err: <nil> Dec 23 14:28:04.831851378 (430h59m4.702934423s) [noise=true , address=45.91.116.6:4975 , piece_id=3MBFI2ILY6VP6XT2PMXFVWG7UA2HYOYYNZASSLX3MLQWTEV7EAKA, seg_pos={0 0}, upload=378343]: Done uploading piece. err: <nil> Dec 23 14:28:04.884609870 (430h59m4.755692915s) [noise=true , address=45.91.116.4:4939 , piece_id=3BWKNBSDTQ4DBJFYXFY3IPWOJ3JTPPDMRO2DRYA5REJ3BERVMFKQ, seg_pos={0 0}, upload=378343]: Done uploading piece. err: <nil> Dec 23 14:28:05.131970658 (430h59m5.003053693s) [noise=true , address=77.90.31.14:4664 , piece_id=TGN5OFP66O6EI654N63FVJSCOXFSIS3AURJGAGYWLSBHVMAOFYFQ, seg_pos={0 0}, upload=378343]: Done uploading piece. err: <nil> Dec 23 14:28:05.195398928 (430h59m5.066481983s) [noise=true , address=145.79.227.51:4410 , piece_id=T5XXHOFBLW3A3OKTDY5CTO74VFCY5ZQJRJPACYF3SCLOTF6E25LA, seg_pos={0 0}, upload=378343]: Done uploading piece. err: <nil> Dec 23 14:28:05.218188272 (430h59m5.089271317s) [noise=true , address=145.79.227.11:4731 , piece_id=JDCFIGBBNHW67SQCYI7PKCKWPUQBCGO3BZROXPHNABIGFFMATMOQ, seg_pos={0 0}, upload=378343]: Done uploading piece. err: <nil> Dec 23 14:28:05.218214873 (430h59m5.089297908s) [seg_pos={0 0}, upload=378343]: Segment reached optimal threshold of 13 pieces. Dec 23 14:28:05.218460901 (430h59m5.089543966s) [noise=true , address=145.79.227.37:4565 , piece_id=RIXR6277D5GYIDPGNMMSJAQLFCMM4P6LWALPUSNFW67BWRXQJ4SQ, seg_pos={0 0}, upload=378343]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nvvNczHtQrSEJuJbErnjiAWa4BRDzBrQmBNbB7RrrqN1JWNHT): protocol: write tcp 188.166.225.170:39840->145.79.227.37:4565: use of closed network connection; write tcp 188.166.225.170:39840->145.79.227.37:4565: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39840->145.79.227.37:4565: use of closed network connection Dec 23 14:28:05.218469708 (430h59m5.089552753s) [noise=true , address=145.79.227.37:4565 , piece_id=RIXR6277D5GYIDPGNMMSJAQLFCMM4P6LWALPUSNFW67BWRXQJ4SQ, seg_pos={0 0}, upload=378343]: Long tail cancellation due to optimal threshold Dec 23 14:28:05.218750874 (430h59m5.089833919s) [noise=true , address=145.79.227.40:4049 , piece_id=3QAECV5EQI3BBHUW7AZXVYWZY6DAOOH6DVP4XAMDXBFJAV34QB5Q, seg_pos={0 0}, upload=378343]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF Dec 23 14:28:05.218770391 (430h59m5.089853436s) [noise=true , address=145.79.227.40:4049 , piece_id=3QAECV5EQI3BBHUW7AZXVYWZY6DAOOH6DVP4XAMDXBFJAV34QB5Q, seg_pos={0 0}, upload=378343]: Long tail cancellation due to optimal threshold Dec 23 14:28:05.219134284 (430h59m5.090217349s) [noise=true , address=145.79.227.52:4445 , piece_id=VIE24CGHGMPQSCGUQKOGNIOMJMZHYD4BRZSGEPMUSPPAXPSFOHZQ, seg_pos={0 0}, upload=378343]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12vXrkhLqCofhR4Fmfqb98gxnqWQF6vMV4q3ATbRvPfcqi82WSo): protocol: expected piece hash; context canceled; EOF Dec 23 14:28:05.219146146 (430h59m5.090229201s) [noise=true , address=145.79.227.52:4445 , piece_id=VIE24CGHGMPQSCGUQKOGNIOMJMZHYD4BRZSGEPMUSPPAXPSFOHZQ, seg_pos={0 0}, upload=378343]: Long tail cancellation due to optimal threshold Dec 23 14:28:05.219173128 (430h59m5.090256173s) [seg_pos={0 0}, upload=378343]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 14:28:05.219417493 (430h59m5.090500538s) [upload=378343]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 14:28:05.219425228 (430h59m5.090508293s) [upload=378343]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 14:28:05.522079636 (430h59m5.393162671s) [upload=378343]: Done uploading segments... Dec 23 14:28:05.522100636 (430h59m5.393183681s) [upload=378343]: Upload finished. err: <nil> Duration: 6.805296101s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.............X.....X............X...........................] duration: 3.798192318s trace: https://jaeger.dev.storj.io/trace/772f273c480c0b2b FAIL (2025-12-23 14:33:58 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.798218347s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [...................X..........X.XX..X............XX..X......] duration: 6.388647218s trace: https://jaeger.dev.storj.io/trace/215296e6cb5de778 FAIL (2025-12-23 14:33:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9, Piece ID: TGN5OFP66O6EI654N63FVJSCOXFSIS3AURJGAGYWLSBHVMAOFYFQ) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: 3MBFI2ILY6VP6XT2PMXFVWG7UA2HYOYYNZASSLX3MLQWTEV7EAKA) context deadline exceeded ======LOGS====== Dec 23 14:28:02.333620097 (430h59m2.204703152s) [upload=378343]: Starting upload Dec 23 14:28:02.333737752 (430h59m2.204820797s) [upload=378343]: Done starting upload Dec 23 14:28:02.333838052 (430h59m2.204921107s) [upload=378343]: Uploading segments... Dec 23 14:28:02.334165316 (430h59m2.205248381s) [upload=378343]: Got next segment. Inline: false Dec 23 14:28:02.334217274 (430h59m2.205300319s) [upload=378343]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 14:28:02.334693501 (430h59m2.205776567s) [upload=378343]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 14:28:02.588168613 (430h59m2.459251658s) [seg_pos={0 0}, upload=378343]: Begin upload segment... Dec 23 14:28:02.588224871 (430h59m2.459307916s) [seg_pos={0 0}, upload=378343]: Done begin upload segment. Dec 23 14:28:02.588264035 (430h59m2.45934707s) [noise=true , address=145.79.227.32:4834 , piece_id=FLPDIEXB34RV3VYR4LZYW63M76C3G2352VTJPF6IHDJSKAY2VSIQ, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.588315313 (430h59m2.459398349s) [noise=true , address=145.79.227.11:4731 , piece_id=JDCFIGBBNHW67SQCYI7PKCKWPUQBCGO3BZROXPHNABIGFFMATMOQ, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.589826190 (430h59m2.460909225s) [noise=true , address=145.79.227.37:4565 , piece_id=RIXR6277D5GYIDPGNMMSJAQLFCMM4P6LWALPUSNFW67BWRXQJ4SQ, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.589974662 (430h59m2.461057708s) [noise=true , address=145.79.227.50:4375 , piece_id=2L6LRC33345CFQ776H4C3RBHD27J3WN5BBB7QUEQ43EQED5LQJRA, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.590200293 (430h59m2.461283328s) [noise=true , address=145.79.227.52:4445 , piece_id=VIE24CGHGMPQSCGUQKOGNIOMJMZHYD4BRZSGEPMUSPPAXPSFOHZQ, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.590571109 (430h59m2.461654154s) [noise=true , address=145.79.227.39:4014 , piece_id=UIYHVARR7QJN7TC45DPH7YMWLOV27EVHFR4HNZXTNGCFCYZ5QWSQ, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.590591117 (430h59m2.461674162s) [noise=true , address=77.90.31.14:4664 , piece_id=TGN5OFP66O6EI654N63FVJSCOXFSIS3AURJGAGYWLSBHVMAOFYFQ, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.591120444 (430h59m2.462203489s) [noise=true , address=145.79.227.38:4978 , piece_id=7M2D3NR342SZVWHS754YCZYVPX2SAKZDUHNV5MIUMG7NQYMJORXA, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.591482514 (430h59m2.462565559s) [noise=true , address=145.79.227.15:4873 , piece_id=RROMZU3GQXWHZ2KAHKJNUXB6JYJW5DOZIMS4RTI2CK4H6GJ4JJPA, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.592004899 (430h59m2.463087944s) [noise=true , address=45.91.116.6:4975 , piece_id=3MBFI2ILY6VP6XT2PMXFVWG7UA2HYOYYNZASSLX3MLQWTEV7EAKA, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.592099669 (430h59m2.463182724s) [noise=true , address=145.79.227.20:4438 , piece_id=OHBU6DZOK7H6MIDZHMXEPL2MDY2OXQSX3VPDIE3RDW2THSMC2IMA, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.593944512 (430h59m2.465027557s) [noise=true , address=145.79.227.16:4910 , piece_id=FDXOMBKKQQPPL3G577PWWCWDJFSGZ3W3Q6OZYANVQ7XSAINJ7MFA, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.597912307 (430h59m2.468995352s) [noise=true , address=145.79.227.29:4943 , piece_id=FF4Q6CY7O4EUFQO5QY7GHSYAAQFHK2WCH7A2BR5B6JGJH4MGHSTA, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.600520093 (430h59m2.471603128s) [noise=true , address=145.79.227.40:4049 , piece_id=3QAECV5EQI3BBHUW7AZXVYWZY6DAOOH6DVP4XAMDXBFJAV34QB5Q, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.602548155 (430h59m2.47363121s) [noise=true , address=45.91.116.4:4939 , piece_id=3BWKNBSDTQ4DBJFYXFY3IPWOJ3JTPPDMRO2DRYA5REJ3BERVMFKQ, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.604654816 (430h59m2.475737861s) [noise=true , address=145.79.227.51:4410 , piece_id=T5XXHOFBLW3A3OKTDY5CTO74VFCY5ZQJRJPACYF3SCLOTF6E25LA, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:04.341061516 (430h59m4.212144561s) [upload=378343]: Next returned nil segment Dec 23 14:28:04.341149894 (430h59m4.212232949s) [upload=378343]: Waiting for error group managing segments... Dec 23 14:28:04.631523518 (430h59m4.502606563s) [noise=true , address=145.79.227.16:4910 , piece_id=FDXOMBKKQQPPL3G577PWWCWDJFSGZ3W3Q6OZYANVQ7XSAINJ7MFA, seg_pos={0 0}, upload=378343]: Done uploading piece. err: <nil> Dec 23 14:28:04.649224493 (430h59m4.520307538s) [noise=true , address=145.79.227.29:4943 , piece_id=FF4Q6CY7O4EUFQO5QY7GHSYAAQFHK2WCH7A2BR5B6JGJH4MGHSTA, seg_pos={0 0}, upload=378343]: Done uploading piece. err: <nil> Dec 23 14:28:04.655099941 (430h59m4.526182986s) [noise=true , address=145.79.227.38:4978 , piece_id=7M2D3NR342SZVWHS754YCZYVPX2SAKZDUHNV5MIUMG7NQYMJORXA, seg_pos={0 0}, upload=378343]: Done uploading piece. err: <nil> Dec 23 14:28:04.663569098 (430h59m4.534652133s) [noise=true , address=145.79.227.15:4873 , piece_id=RROMZU3GQXWHZ2KAHKJNUXB6JYJW5DOZIMS4RTI2CK4H6GJ4JJPA, seg_pos={0 0}, upload=378343]: Done uploading piece. err: <nil> Dec 23 14:28:04.784039696 (430h59m4.655122741s) [noise=true , address=145.79.227.39:4014 , piece_id=UIYHVARR7QJN7TC45DPH7YMWLOV27EVHFR4HNZXTNGCFCYZ5QWSQ, seg_pos={0 0}, upload=378343]: Done uploading piece. err: <nil> Dec 23 14:28:04.789452723 (430h59m4.660535769s) [noise=true , address=145.79.227.50:4375 , piece_id=2L6LRC33345CFQ776H4C3RBHD27J3WN5BBB7QUEQ43EQED5LQJRA, seg_pos={0 0}, upload=378343]: Done uploading piece. err: <nil> Dec 23 14:28:04.809692453 (430h59m4.680775488s) [noise=true , address=145.79.227.32:4834 , piece_id=FLPDIEXB34RV3VYR4LZYW63M76C3G2352VTJPF6IHDJSKAY2VSIQ, seg_pos={0 0}, upload=378343]: Done uploading piece. err: <nil> Dec 23 14:28:04.814157645 (430h59m4.68524069s) [noise=true , address=145.79.227.20:4438 , piece_id=OHBU6DZOK7H6MIDZHMXEPL2MDY2OXQSX3VPDIE3RDW2THSMC2IMA, seg_pos={0 0}, upload=378343]: Done uploading piece. err: <nil> Dec 23 14:28:04.831851378 (430h59m4.702934423s) [noise=true , address=45.91.116.6:4975 , piece_id=3MBFI2ILY6VP6XT2PMXFVWG7UA2HYOYYNZASSLX3MLQWTEV7EAKA, seg_pos={0 0}, upload=378343]: Done uploading piece. err: <nil> Dec 23 14:28:04.884609870 (430h59m4.755692915s) [noise=true , address=45.91.116.4:4939 , piece_id=3BWKNBSDTQ4DBJFYXFY3IPWOJ3JTPPDMRO2DRYA5REJ3BERVMFKQ, seg_pos={0 0}, upload=378343]: Done uploading piece. err: <nil> Dec 23 14:28:05.131970658 (430h59m5.003053693s) [noise=true , address=77.90.31.14:4664 , piece_id=TGN5OFP66O6EI654N63FVJSCOXFSIS3AURJGAGYWLSBHVMAOFYFQ, seg_pos={0 0}, upload=378343]: Done uploading piece. err: <nil> Dec 23 14:28:05.195398928 (430h59m5.066481983s) [noise=true , address=145.79.227.51:4410 , piece_id=T5XXHOFBLW3A3OKTDY5CTO74VFCY5ZQJRJPACYF3SCLOTF6E25LA, seg_pos={0 0}, upload=378343]: Done uploading piece. err: <nil> Dec 23 14:28:05.218188272 (430h59m5.089271317s) [noise=true , address=145.79.227.11:4731 , piece_id=JDCFIGBBNHW67SQCYI7PKCKWPUQBCGO3BZROXPHNABIGFFMATMOQ, seg_pos={0 0}, upload=378343]: Done uploading piece. err: <nil> Dec 23 14:28:05.218214873 (430h59m5.089297908s) [seg_pos={0 0}, upload=378343]: Segment reached optimal threshold of 13 pieces. Dec 23 14:28:05.218460901 (430h59m5.089543966s) [noise=true , address=145.79.227.37:4565 , piece_id=RIXR6277D5GYIDPGNMMSJAQLFCMM4P6LWALPUSNFW67BWRXQJ4SQ, seg_pos={0 0}, upload=378343]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nvvNczHtQrSEJuJbErnjiAWa4BRDzBrQmBNbB7RrrqN1JWNHT): protocol: write tcp 188.166.225.170:39840->145.79.227.37:4565: use of closed network connection; write tcp 188.166.225.170:39840->145.79.227.37:4565: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39840->145.79.227.37:4565: use of closed network connection Dec 23 14:28:05.218469708 (430h59m5.089552753s) [noise=true , address=145.79.227.37:4565 , piece_id=RIXR6277D5GYIDPGNMMSJAQLFCMM4P6LWALPUSNFW67BWRXQJ4SQ, seg_pos={0 0}, upload=378343]: Long tail cancellation due to optimal threshold Dec 23 14:28:05.218750874 (430h59m5.089833919s) [noise=true , address=145.79.227.40:4049 , piece_id=3QAECV5EQI3BBHUW7AZXVYWZY6DAOOH6DVP4XAMDXBFJAV34QB5Q, seg_pos={0 0}, upload=378343]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF Dec 23 14:28:05.218770391 (430h59m5.089853436s) [noise=true , address=145.79.227.40:4049 , piece_id=3QAECV5EQI3BBHUW7AZXVYWZY6DAOOH6DVP4XAMDXBFJAV34QB5Q, seg_pos={0 0}, upload=378343]: Long tail cancellation due to optimal threshold Dec 23 14:28:05.219134284 (430h59m5.090217349s) [noise=true , address=145.79.227.52:4445 , piece_id=VIE24CGHGMPQSCGUQKOGNIOMJMZHYD4BRZSGEPMUSPPAXPSFOHZQ, seg_pos={0 0}, upload=378343]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12vXrkhLqCofhR4Fmfqb98gxnqWQF6vMV4q3ATbRvPfcqi82WSo): protocol: expected piece hash; context canceled; EOF Dec 23 14:28:05.219146146 (430h59m5.090229201s) [noise=true , address=145.79.227.52:4445 , piece_id=VIE24CGHGMPQSCGUQKOGNIOMJMZHYD4BRZSGEPMUSPPAXPSFOHZQ, seg_pos={0 0}, upload=378343]: Long tail cancellation due to optimal threshold Dec 23 14:28:05.219173128 (430h59m5.090256173s) [seg_pos={0 0}, upload=378343]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 14:28:05.219417493 (430h59m5.090500538s) [upload=378343]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 14:28:05.219425228 (430h59m5.090508293s) [upload=378343]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 14:28:05.522079636 (430h59m5.393162671s) [upload=378343]: Done uploading segments... Dec 23 14:28:05.522100636 (430h59m5.393183681s) [upload=378343]: Upload finished. err: <nil> Duration: 6.388670572s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [..............X.....X............X..........................] duration: 3.561323248s trace: https://jaeger.dev.storj.io/trace/772f273c480c0b2b FAIL (2025-12-23 14:33:16 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.561355729s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [....................X..........X.XX..X............XX..X.....] duration: 7.372174188s trace: https://jaeger.dev.storj.io/trace/215296e6cb5de778 FAIL (2025-12-23 14:32:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9, Piece ID: TGN5OFP66O6EI654N63FVJSCOXFSIS3AURJGAGYWLSBHVMAOFYFQ) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: 3MBFI2ILY6VP6XT2PMXFVWG7UA2HYOYYNZASSLX3MLQWTEV7EAKA) context deadline exceeded ======LOGS====== Dec 23 14:28:02.333620097 (430h59m2.204703152s) [upload=378343]: Starting upload Dec 23 14:28:02.333737752 (430h59m2.204820797s) [upload=378343]: Done starting upload Dec 23 14:28:02.333838052 (430h59m2.204921107s) [upload=378343]: Uploading segments... Dec 23 14:28:02.334165316 (430h59m2.205248381s) [upload=378343]: Got next segment. Inline: false Dec 23 14:28:02.334217274 (430h59m2.205300319s) [upload=378343]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 14:28:02.334693501 (430h59m2.205776567s) [upload=378343]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 14:28:02.588168613 (430h59m2.459251658s) [seg_pos={0 0}, upload=378343]: Begin upload segment... Dec 23 14:28:02.588224871 (430h59m2.459307916s) [seg_pos={0 0}, upload=378343]: Done begin upload segment. Dec 23 14:28:02.588264035 (430h59m2.45934707s) [noise=true , address=145.79.227.32:4834 , piece_id=FLPDIEXB34RV3VYR4LZYW63M76C3G2352VTJPF6IHDJSKAY2VSIQ, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.588315313 (430h59m2.459398349s) [noise=true , address=145.79.227.11:4731 , piece_id=JDCFIGBBNHW67SQCYI7PKCKWPUQBCGO3BZROXPHNABIGFFMATMOQ, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.589826190 (430h59m2.460909225s) [noise=true , address=145.79.227.37:4565 , piece_id=RIXR6277D5GYIDPGNMMSJAQLFCMM4P6LWALPUSNFW67BWRXQJ4SQ, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.589974662 (430h59m2.461057708s) [noise=true , address=145.79.227.50:4375 , piece_id=2L6LRC33345CFQ776H4C3RBHD27J3WN5BBB7QUEQ43EQED5LQJRA, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.590200293 (430h59m2.461283328s) [noise=true , address=145.79.227.52:4445 , piece_id=VIE24CGHGMPQSCGUQKOGNIOMJMZHYD4BRZSGEPMUSPPAXPSFOHZQ, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.590571109 (430h59m2.461654154s) [noise=true , address=145.79.227.39:4014 , piece_id=UIYHVARR7QJN7TC45DPH7YMWLOV27EVHFR4HNZXTNGCFCYZ5QWSQ, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.590591117 (430h59m2.461674162s) [noise=true , address=77.90.31.14:4664 , piece_id=TGN5OFP66O6EI654N63FVJSCOXFSIS3AURJGAGYWLSBHVMAOFYFQ, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.591120444 (430h59m2.462203489s) [noise=true , address=145.79.227.38:4978 , piece_id=7M2D3NR342SZVWHS754YCZYVPX2SAKZDUHNV5MIUMG7NQYMJORXA, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.591482514 (430h59m2.462565559s) [noise=true , address=145.79.227.15:4873 , piece_id=RROMZU3GQXWHZ2KAHKJNUXB6JYJW5DOZIMS4RTI2CK4H6GJ4JJPA, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.592004899 (430h59m2.463087944s) [noise=true , address=45.91.116.6:4975 , piece_id=3MBFI2ILY6VP6XT2PMXFVWG7UA2HYOYYNZASSLX3MLQWTEV7EAKA, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.592099669 (430h59m2.463182724s) [noise=true , address=145.79.227.20:4438 , piece_id=OHBU6DZOK7H6MIDZHMXEPL2MDY2OXQSX3VPDIE3RDW2THSMC2IMA, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.593944512 (430h59m2.465027557s) [noise=true , address=145.79.227.16:4910 , piece_id=FDXOMBKKQQPPL3G577PWWCWDJFSGZ3W3Q6OZYANVQ7XSAINJ7MFA, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.597912307 (430h59m2.468995352s) [noise=true , address=145.79.227.29:4943 , piece_id=FF4Q6CY7O4EUFQO5QY7GHSYAAQFHK2WCH7A2BR5B6JGJH4MGHSTA, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.600520093 (430h59m2.471603128s) [noise=true , address=145.79.227.40:4049 , piece_id=3QAECV5EQI3BBHUW7AZXVYWZY6DAOOH6DVP4XAMDXBFJAV34QB5Q, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.602548155 (430h59m2.47363121s) [noise=true , address=45.91.116.4:4939 , piece_id=3BWKNBSDTQ4DBJFYXFY3IPWOJ3JTPPDMRO2DRYA5REJ3BERVMFKQ, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.604654816 (430h59m2.475737861s) [noise=true , address=145.79.227.51:4410 , piece_id=T5XXHOFBLW3A3OKTDY5CTO74VFCY5ZQJRJPACYF3SCLOTF6E25LA, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:04.341061516 (430h59m4.212144561s) [upload=378343]: Next returned nil segment Dec 23 14:28:04.341149894 (430h59m4.212232949s) [upload=378343]: Waiting for error group managing segments... Dec 23 14:28:04.631523518 (430h59m4.502606563s) [noise=true , address=145.79.227.16:4910 , piece_id=FDXOMBKKQQPPL3G577PWWCWDJFSGZ3W3Q6OZYANVQ7XSAINJ7MFA, seg_pos={0 0}, upload=378343]: Done uploading piece. err: <nil> Dec 23 14:28:04.649224493 (430h59m4.520307538s) [noise=true , address=145.79.227.29:4943 , piece_id=FF4Q6CY7O4EUFQO5QY7GHSYAAQFHK2WCH7A2BR5B6JGJH4MGHSTA, seg_pos={0 0}, upload=378343]: Done uploading piece. err: <nil> Dec 23 14:28:04.655099941 (430h59m4.526182986s) [noise=true , address=145.79.227.38:4978 , piece_id=7M2D3NR342SZVWHS754YCZYVPX2SAKZDUHNV5MIUMG7NQYMJORXA, seg_pos={0 0}, upload=378343]: Done uploading piece. err: <nil> Dec 23 14:28:04.663569098 (430h59m4.534652133s) [noise=true , address=145.79.227.15:4873 , piece_id=RROMZU3GQXWHZ2KAHKJNUXB6JYJW5DOZIMS4RTI2CK4H6GJ4JJPA, seg_pos={0 0}, upload=378343]: Done uploading piece. err: <nil> Dec 23 14:28:04.784039696 (430h59m4.655122741s) [noise=true , address=145.79.227.39:4014 , piece_id=UIYHVARR7QJN7TC45DPH7YMWLOV27EVHFR4HNZXTNGCFCYZ5QWSQ, seg_pos={0 0}, upload=378343]: Done uploading piece. err: <nil> Dec 23 14:28:04.789452723 (430h59m4.660535769s) [noise=true , address=145.79.227.50:4375 , piece_id=2L6LRC33345CFQ776H4C3RBHD27J3WN5BBB7QUEQ43EQED5LQJRA, seg_pos={0 0}, upload=378343]: Done uploading piece. err: <nil> Dec 23 14:28:04.809692453 (430h59m4.680775488s) [noise=true , address=145.79.227.32:4834 , piece_id=FLPDIEXB34RV3VYR4LZYW63M76C3G2352VTJPF6IHDJSKAY2VSIQ, seg_pos={0 0}, upload=378343]: Done uploading piece. err: <nil> Dec 23 14:28:04.814157645 (430h59m4.68524069s) [noise=true , address=145.79.227.20:4438 , piece_id=OHBU6DZOK7H6MIDZHMXEPL2MDY2OXQSX3VPDIE3RDW2THSMC2IMA, seg_pos={0 0}, upload=378343]: Done uploading piece. err: <nil> Dec 23 14:28:04.831851378 (430h59m4.702934423s) [noise=true , address=45.91.116.6:4975 , piece_id=3MBFI2ILY6VP6XT2PMXFVWG7UA2HYOYYNZASSLX3MLQWTEV7EAKA, seg_pos={0 0}, upload=378343]: Done uploading piece. err: <nil> Dec 23 14:28:04.884609870 (430h59m4.755692915s) [noise=true , address=45.91.116.4:4939 , piece_id=3BWKNBSDTQ4DBJFYXFY3IPWOJ3JTPPDMRO2DRYA5REJ3BERVMFKQ, seg_pos={0 0}, upload=378343]: Done uploading piece. err: <nil> Dec 23 14:28:05.131970658 (430h59m5.003053693s) [noise=true , address=77.90.31.14:4664 , piece_id=TGN5OFP66O6EI654N63FVJSCOXFSIS3AURJGAGYWLSBHVMAOFYFQ, seg_pos={0 0}, upload=378343]: Done uploading piece. err: <nil> Dec 23 14:28:05.195398928 (430h59m5.066481983s) [noise=true , address=145.79.227.51:4410 , piece_id=T5XXHOFBLW3A3OKTDY5CTO74VFCY5ZQJRJPACYF3SCLOTF6E25LA, seg_pos={0 0}, upload=378343]: Done uploading piece. err: <nil> Dec 23 14:28:05.218188272 (430h59m5.089271317s) [noise=true , address=145.79.227.11:4731 , piece_id=JDCFIGBBNHW67SQCYI7PKCKWPUQBCGO3BZROXPHNABIGFFMATMOQ, seg_pos={0 0}, upload=378343]: Done uploading piece. err: <nil> Dec 23 14:28:05.218214873 (430h59m5.089297908s) [seg_pos={0 0}, upload=378343]: Segment reached optimal threshold of 13 pieces. Dec 23 14:28:05.218460901 (430h59m5.089543966s) [noise=true , address=145.79.227.37:4565 , piece_id=RIXR6277D5GYIDPGNMMSJAQLFCMM4P6LWALPUSNFW67BWRXQJ4SQ, seg_pos={0 0}, upload=378343]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nvvNczHtQrSEJuJbErnjiAWa4BRDzBrQmBNbB7RrrqN1JWNHT): protocol: write tcp 188.166.225.170:39840->145.79.227.37:4565: use of closed network connection; write tcp 188.166.225.170:39840->145.79.227.37:4565: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39840->145.79.227.37:4565: use of closed network connection Dec 23 14:28:05.218469708 (430h59m5.089552753s) [noise=true , address=145.79.227.37:4565 , piece_id=RIXR6277D5GYIDPGNMMSJAQLFCMM4P6LWALPUSNFW67BWRXQJ4SQ, seg_pos={0 0}, upload=378343]: Long tail cancellation due to optimal threshold Dec 23 14:28:05.218750874 (430h59m5.089833919s) [noise=true , address=145.79.227.40:4049 , piece_id=3QAECV5EQI3BBHUW7AZXVYWZY6DAOOH6DVP4XAMDXBFJAV34QB5Q, seg_pos={0 0}, upload=378343]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF Dec 23 14:28:05.218770391 (430h59m5.089853436s) [noise=true , address=145.79.227.40:4049 , piece_id=3QAECV5EQI3BBHUW7AZXVYWZY6DAOOH6DVP4XAMDXBFJAV34QB5Q, seg_pos={0 0}, upload=378343]: Long tail cancellation due to optimal threshold Dec 23 14:28:05.219134284 (430h59m5.090217349s) [noise=true , address=145.79.227.52:4445 , piece_id=VIE24CGHGMPQSCGUQKOGNIOMJMZHYD4BRZSGEPMUSPPAXPSFOHZQ, seg_pos={0 0}, upload=378343]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12vXrkhLqCofhR4Fmfqb98gxnqWQF6vMV4q3ATbRvPfcqi82WSo): protocol: expected piece hash; context canceled; EOF Dec 23 14:28:05.219146146 (430h59m5.090229201s) [noise=true , address=145.79.227.52:4445 , piece_id=VIE24CGHGMPQSCGUQKOGNIOMJMZHYD4BRZSGEPMUSPPAXPSFOHZQ, seg_pos={0 0}, upload=378343]: Long tail cancellation due to optimal threshold Dec 23 14:28:05.219173128 (430h59m5.090256173s) [seg_pos={0 0}, upload=378343]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 14:28:05.219417493 (430h59m5.090500538s) [upload=378343]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 14:28:05.219425228 (430h59m5.090508293s) [upload=378343]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 14:28:05.522079636 (430h59m5.393162671s) [upload=378343]: Done uploading segments... Dec 23 14:28:05.522100636 (430h59m5.393183681s) [upload=378343]: Upload finished. err: <nil> Duration: 7.372200809s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [...............X.....X............X.........................] duration: 3.601359925s trace: https://jaeger.dev.storj.io/trace/772f273c480c0b2b FAIL (2025-12-23 14:32:19 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.601390752s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.....................X..........X.XX..X............XX..X....] duration: 7.164469763s trace: https://jaeger.dev.storj.io/trace/215296e6cb5de778 FAIL (2025-12-23 14:31:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9, Piece ID: TGN5OFP66O6EI654N63FVJSCOXFSIS3AURJGAGYWLSBHVMAOFYFQ) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: 3MBFI2ILY6VP6XT2PMXFVWG7UA2HYOYYNZASSLX3MLQWTEV7EAKA) context deadline exceeded ======LOGS====== Dec 23 14:28:02.333620097 (430h59m2.204703152s) [upload=378343]: Starting upload Dec 23 14:28:02.333737752 (430h59m2.204820797s) [upload=378343]: Done starting upload Dec 23 14:28:02.333838052 (430h59m2.204921107s) [upload=378343]: Uploading segments... Dec 23 14:28:02.334165316 (430h59m2.205248381s) [upload=378343]: Got next segment. Inline: false Dec 23 14:28:02.334217274 (430h59m2.205300319s) [upload=378343]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 14:28:02.334693501 (430h59m2.205776567s) [upload=378343]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 14:28:02.588168613 (430h59m2.459251658s) [seg_pos={0 0}, upload=378343]: Begin upload segment... Dec 23 14:28:02.588224871 (430h59m2.459307916s) [seg_pos={0 0}, upload=378343]: Done begin upload segment. Dec 23 14:28:02.588264035 (430h59m2.45934707s) [noise=true , address=145.79.227.32:4834 , piece_id=FLPDIEXB34RV3VYR4LZYW63M76C3G2352VTJPF6IHDJSKAY2VSIQ, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.588315313 (430h59m2.459398349s) [noise=true , address=145.79.227.11:4731 , piece_id=JDCFIGBBNHW67SQCYI7PKCKWPUQBCGO3BZROXPHNABIGFFMATMOQ, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.589826190 (430h59m2.460909225s) [noise=true , address=145.79.227.37:4565 , piece_id=RIXR6277D5GYIDPGNMMSJAQLFCMM4P6LWALPUSNFW67BWRXQJ4SQ, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.589974662 (430h59m2.461057708s) [noise=true , address=145.79.227.50:4375 , piece_id=2L6LRC33345CFQ776H4C3RBHD27J3WN5BBB7QUEQ43EQED5LQJRA, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.590200293 (430h59m2.461283328s) [noise=true , address=145.79.227.52:4445 , piece_id=VIE24CGHGMPQSCGUQKOGNIOMJMZHYD4BRZSGEPMUSPPAXPSFOHZQ, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.590571109 (430h59m2.461654154s) [noise=true , address=145.79.227.39:4014 , piece_id=UIYHVARR7QJN7TC45DPH7YMWLOV27EVHFR4HNZXTNGCFCYZ5QWSQ, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.590591117 (430h59m2.461674162s) [noise=true , address=77.90.31.14:4664 , piece_id=TGN5OFP66O6EI654N63FVJSCOXFSIS3AURJGAGYWLSBHVMAOFYFQ, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.591120444 (430h59m2.462203489s) [noise=true , address=145.79.227.38:4978 , piece_id=7M2D3NR342SZVWHS754YCZYVPX2SAKZDUHNV5MIUMG7NQYMJORXA, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.591482514 (430h59m2.462565559s) [noise=true , address=145.79.227.15:4873 , piece_id=RROMZU3GQXWHZ2KAHKJNUXB6JYJW5DOZIMS4RTI2CK4H6GJ4JJPA, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.592004899 (430h59m2.463087944s) [noise=true , address=45.91.116.6:4975 , piece_id=3MBFI2ILY6VP6XT2PMXFVWG7UA2HYOYYNZASSLX3MLQWTEV7EAKA, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.592099669 (430h59m2.463182724s) [noise=true , address=145.79.227.20:4438 , piece_id=OHBU6DZOK7H6MIDZHMXEPL2MDY2OXQSX3VPDIE3RDW2THSMC2IMA, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.593944512 (430h59m2.465027557s) [noise=true , address=145.79.227.16:4910 , piece_id=FDXOMBKKQQPPL3G577PWWCWDJFSGZ3W3Q6OZYANVQ7XSAINJ7MFA, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.597912307 (430h59m2.468995352s) [noise=true , address=145.79.227.29:4943 , piece_id=FF4Q6CY7O4EUFQO5QY7GHSYAAQFHK2WCH7A2BR5B6JGJH4MGHSTA, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.600520093 (430h59m2.471603128s) [noise=true , address=145.79.227.40:4049 , piece_id=3QAECV5EQI3BBHUW7AZXVYWZY6DAOOH6DVP4XAMDXBFJAV34QB5Q, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.602548155 (430h59m2.47363121s) [noise=true , address=45.91.116.4:4939 , piece_id=3BWKNBSDTQ4DBJFYXFY3IPWOJ3JTPPDMRO2DRYA5REJ3BERVMFKQ, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.604654816 (430h59m2.475737861s) [noise=true , address=145.79.227.51:4410 , piece_id=T5XXHOFBLW3A3OKTDY5CTO74VFCY5ZQJRJPACYF3SCLOTF6E25LA, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:04.341061516 (430h59m4.212144561s) [upload=378343]: Next returned nil segment Dec 23 14:28:04.341149894 (430h59m4.212232949s) [upload=378343]: Waiting for error group managing segments... Dec 23 14:28:04.631523518 (430h59m4.502606563s) [noise=true , address=145.79.227.16:4910 , piece_id=FDXOMBKKQQPPL3G577PWWCWDJFSGZ3W3Q6OZYANVQ7XSAINJ7MFA, seg_pos={0 0}, upload=378343]: Done uploading piece. err: <nil> Dec 23 14:28:04.649224493 (430h59m4.520307538s) [noise=true , address=145.79.227.29:4943 , piece_id=FF4Q6CY7O4EUFQO5QY7GHSYAAQFHK2WCH7A2BR5B6JGJH4MGHSTA, seg_pos={0 0}, upload=378343]: Done uploading piece. err: <nil> Dec 23 14:28:04.655099941 (430h59m4.526182986s) [noise=true , address=145.79.227.38:4978 , piece_id=7M2D3NR342SZVWHS754YCZYVPX2SAKZDUHNV5MIUMG7NQYMJORXA, seg_pos={0 0}, upload=378343]: Done uploading piece. err: <nil> Dec 23 14:28:04.663569098 (430h59m4.534652133s) [noise=true , address=145.79.227.15:4873 , piece_id=RROMZU3GQXWHZ2KAHKJNUXB6JYJW5DOZIMS4RTI2CK4H6GJ4JJPA, seg_pos={0 0}, upload=378343]: Done uploading piece. err: <nil> Dec 23 14:28:04.784039696 (430h59m4.655122741s) [noise=true , address=145.79.227.39:4014 , piece_id=UIYHVARR7QJN7TC45DPH7YMWLOV27EVHFR4HNZXTNGCFCYZ5QWSQ, seg_pos={0 0}, upload=378343]: Done uploading piece. err: <nil> Dec 23 14:28:04.789452723 (430h59m4.660535769s) [noise=true , address=145.79.227.50:4375 , piece_id=2L6LRC33345CFQ776H4C3RBHD27J3WN5BBB7QUEQ43EQED5LQJRA, seg_pos={0 0}, upload=378343]: Done uploading piece. err: <nil> Dec 23 14:28:04.809692453 (430h59m4.680775488s) [noise=true , address=145.79.227.32:4834 , piece_id=FLPDIEXB34RV3VYR4LZYW63M76C3G2352VTJPF6IHDJSKAY2VSIQ, seg_pos={0 0}, upload=378343]: Done uploading piece. err: <nil> Dec 23 14:28:04.814157645 (430h59m4.68524069s) [noise=true , address=145.79.227.20:4438 , piece_id=OHBU6DZOK7H6MIDZHMXEPL2MDY2OXQSX3VPDIE3RDW2THSMC2IMA, seg_pos={0 0}, upload=378343]: Done uploading piece. err: <nil> Dec 23 14:28:04.831851378 (430h59m4.702934423s) [noise=true , address=45.91.116.6:4975 , piece_id=3MBFI2ILY6VP6XT2PMXFVWG7UA2HYOYYNZASSLX3MLQWTEV7EAKA, seg_pos={0 0}, upload=378343]: Done uploading piece. err: <nil> Dec 23 14:28:04.884609870 (430h59m4.755692915s) [noise=true , address=45.91.116.4:4939 , piece_id=3BWKNBSDTQ4DBJFYXFY3IPWOJ3JTPPDMRO2DRYA5REJ3BERVMFKQ, seg_pos={0 0}, upload=378343]: Done uploading piece. err: <nil> Dec 23 14:28:05.131970658 (430h59m5.003053693s) [noise=true , address=77.90.31.14:4664 , piece_id=TGN5OFP66O6EI654N63FVJSCOXFSIS3AURJGAGYWLSBHVMAOFYFQ, seg_pos={0 0}, upload=378343]: Done uploading piece. err: <nil> Dec 23 14:28:05.195398928 (430h59m5.066481983s) [noise=true , address=145.79.227.51:4410 , piece_id=T5XXHOFBLW3A3OKTDY5CTO74VFCY5ZQJRJPACYF3SCLOTF6E25LA, seg_pos={0 0}, upload=378343]: Done uploading piece. err: <nil> Dec 23 14:28:05.218188272 (430h59m5.089271317s) [noise=true , address=145.79.227.11:4731 , piece_id=JDCFIGBBNHW67SQCYI7PKCKWPUQBCGO3BZROXPHNABIGFFMATMOQ, seg_pos={0 0}, upload=378343]: Done uploading piece. err: <nil> Dec 23 14:28:05.218214873 (430h59m5.089297908s) [seg_pos={0 0}, upload=378343]: Segment reached optimal threshold of 13 pieces. Dec 23 14:28:05.218460901 (430h59m5.089543966s) [noise=true , address=145.79.227.37:4565 , piece_id=RIXR6277D5GYIDPGNMMSJAQLFCMM4P6LWALPUSNFW67BWRXQJ4SQ, seg_pos={0 0}, upload=378343]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nvvNczHtQrSEJuJbErnjiAWa4BRDzBrQmBNbB7RrrqN1JWNHT): protocol: write tcp 188.166.225.170:39840->145.79.227.37:4565: use of closed network connection; write tcp 188.166.225.170:39840->145.79.227.37:4565: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39840->145.79.227.37:4565: use of closed network connection Dec 23 14:28:05.218469708 (430h59m5.089552753s) [noise=true , address=145.79.227.37:4565 , piece_id=RIXR6277D5GYIDPGNMMSJAQLFCMM4P6LWALPUSNFW67BWRXQJ4SQ, seg_pos={0 0}, upload=378343]: Long tail cancellation due to optimal threshold Dec 23 14:28:05.218750874 (430h59m5.089833919s) [noise=true , address=145.79.227.40:4049 , piece_id=3QAECV5EQI3BBHUW7AZXVYWZY6DAOOH6DVP4XAMDXBFJAV34QB5Q, seg_pos={0 0}, upload=378343]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF Dec 23 14:28:05.218770391 (430h59m5.089853436s) [noise=true , address=145.79.227.40:4049 , piece_id=3QAECV5EQI3BBHUW7AZXVYWZY6DAOOH6DVP4XAMDXBFJAV34QB5Q, seg_pos={0 0}, upload=378343]: Long tail cancellation due to optimal threshold Dec 23 14:28:05.219134284 (430h59m5.090217349s) [noise=true , address=145.79.227.52:4445 , piece_id=VIE24CGHGMPQSCGUQKOGNIOMJMZHYD4BRZSGEPMUSPPAXPSFOHZQ, seg_pos={0 0}, upload=378343]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12vXrkhLqCofhR4Fmfqb98gxnqWQF6vMV4q3ATbRvPfcqi82WSo): protocol: expected piece hash; context canceled; EOF Dec 23 14:28:05.219146146 (430h59m5.090229201s) [noise=true , address=145.79.227.52:4445 , piece_id=VIE24CGHGMPQSCGUQKOGNIOMJMZHYD4BRZSGEPMUSPPAXPSFOHZQ, seg_pos={0 0}, upload=378343]: Long tail cancellation due to optimal threshold Dec 23 14:28:05.219173128 (430h59m5.090256173s) [seg_pos={0 0}, upload=378343]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 14:28:05.219417493 (430h59m5.090500538s) [upload=378343]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 14:28:05.219425228 (430h59m5.090508293s) [upload=378343]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 14:28:05.522079636 (430h59m5.393162671s) [upload=378343]: Done uploading segments... Dec 23 14:28:05.522100636 (430h59m5.393183681s) [upload=378343]: Upload finished. err: <nil> Duration: 7.164495093s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [X...............X.....X............X........................] duration: 3.699970275s trace: https://jaeger.dev.storj.io/trace/772f273c480c0b2b FAIL (2025-12-23 14:31:06 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.69999885s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [......................X..........X.XX..X............XX..X...] duration: 13.307355515s trace: https://jaeger.dev.storj.io/trace/215296e6cb5de778 FAIL (2025-12-23 14:30:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9, Piece ID: TGN5OFP66O6EI654N63FVJSCOXFSIS3AURJGAGYWLSBHVMAOFYFQ) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: 3MBFI2ILY6VP6XT2PMXFVWG7UA2HYOYYNZASSLX3MLQWTEV7EAKA) context deadline exceeded ======LOGS====== Dec 23 14:28:02.333620097 (430h59m2.204703152s) [upload=378343]: Starting upload Dec 23 14:28:02.333737752 (430h59m2.204820797s) [upload=378343]: Done starting upload Dec 23 14:28:02.333838052 (430h59m2.204921107s) [upload=378343]: Uploading segments... Dec 23 14:28:02.334165316 (430h59m2.205248381s) [upload=378343]: Got next segment. Inline: false Dec 23 14:28:02.334217274 (430h59m2.205300319s) [upload=378343]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 14:28:02.334693501 (430h59m2.205776567s) [upload=378343]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 14:28:02.588168613 (430h59m2.459251658s) [seg_pos={0 0}, upload=378343]: Begin upload segment... Dec 23 14:28:02.588224871 (430h59m2.459307916s) [seg_pos={0 0}, upload=378343]: Done begin upload segment. Dec 23 14:28:02.588264035 (430h59m2.45934707s) [noise=true , address=145.79.227.32:4834 , piece_id=FLPDIEXB34RV3VYR4LZYW63M76C3G2352VTJPF6IHDJSKAY2VSIQ, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.588315313 (430h59m2.459398349s) [noise=true , address=145.79.227.11:4731 , piece_id=JDCFIGBBNHW67SQCYI7PKCKWPUQBCGO3BZROXPHNABIGFFMATMOQ, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.589826190 (430h59m2.460909225s) [noise=true , address=145.79.227.37:4565 , piece_id=RIXR6277D5GYIDPGNMMSJAQLFCMM4P6LWALPUSNFW67BWRXQJ4SQ, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.589974662 (430h59m2.461057708s) [noise=true , address=145.79.227.50:4375 , piece_id=2L6LRC33345CFQ776H4C3RBHD27J3WN5BBB7QUEQ43EQED5LQJRA, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.590200293 (430h59m2.461283328s) [noise=true , address=145.79.227.52:4445 , piece_id=VIE24CGHGMPQSCGUQKOGNIOMJMZHYD4BRZSGEPMUSPPAXPSFOHZQ, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.590571109 (430h59m2.461654154s) [noise=true , address=145.79.227.39:4014 , piece_id=UIYHVARR7QJN7TC45DPH7YMWLOV27EVHFR4HNZXTNGCFCYZ5QWSQ, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.590591117 (430h59m2.461674162s) [noise=true , address=77.90.31.14:4664 , piece_id=TGN5OFP66O6EI654N63FVJSCOXFSIS3AURJGAGYWLSBHVMAOFYFQ, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.591120444 (430h59m2.462203489s) [noise=true , address=145.79.227.38:4978 , piece_id=7M2D3NR342SZVWHS754YCZYVPX2SAKZDUHNV5MIUMG7NQYMJORXA, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.591482514 (430h59m2.462565559s) [noise=true , address=145.79.227.15:4873 , piece_id=RROMZU3GQXWHZ2KAHKJNUXB6JYJW5DOZIMS4RTI2CK4H6GJ4JJPA, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.592004899 (430h59m2.463087944s) [noise=true , address=45.91.116.6:4975 , piece_id=3MBFI2ILY6VP6XT2PMXFVWG7UA2HYOYYNZASSLX3MLQWTEV7EAKA, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.592099669 (430h59m2.463182724s) [noise=true , address=145.79.227.20:4438 , piece_id=OHBU6DZOK7H6MIDZHMXEPL2MDY2OXQSX3VPDIE3RDW2THSMC2IMA, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.593944512 (430h59m2.465027557s) [noise=true , address=145.79.227.16:4910 , piece_id=FDXOMBKKQQPPL3G577PWWCWDJFSGZ3W3Q6OZYANVQ7XSAINJ7MFA, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.597912307 (430h59m2.468995352s) [noise=true , address=145.79.227.29:4943 , piece_id=FF4Q6CY7O4EUFQO5QY7GHSYAAQFHK2WCH7A2BR5B6JGJH4MGHSTA, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.600520093 (430h59m2.471603128s) [noise=true , address=145.79.227.40:4049 , piece_id=3QAECV5EQI3BBHUW7AZXVYWZY6DAOOH6DVP4XAMDXBFJAV34QB5Q, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.602548155 (430h59m2.47363121s) [noise=true , address=45.91.116.4:4939 , piece_id=3BWKNBSDTQ4DBJFYXFY3IPWOJ3JTPPDMRO2DRYA5REJ3BERVMFKQ, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.604654816 (430h59m2.475737861s) [noise=true , address=145.79.227.51:4410 , piece_id=T5XXHOFBLW3A3OKTDY5CTO74VFCY5ZQJRJPACYF3SCLOTF6E25LA, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:04.341061516 (430h59m4.212144561s) [upload=378343]: Next returned nil segment Dec 23 14:28:04.341149894 (430h59m4.212232949s) [upload=378343]: Waiting for error group managing segments... Dec 23 14:28:04.631523518 (430h59m4.502606563s) [noise=true , address=145.79.227.16:4910 , piece_id=FDXOMBKKQQPPL3G577PWWCWDJFSGZ3W3Q6OZYANVQ7XSAINJ7MFA, seg_pos={0 0}, upload=378343]: Done uploading piece. err: <nil> Dec 23 14:28:04.649224493 (430h59m4.520307538s) [noise=true , address=145.79.227.29:4943 , piece_id=FF4Q6CY7O4EUFQO5QY7GHSYAAQFHK2WCH7A2BR5B6JGJH4MGHSTA, seg_pos={0 0}, upload=378343]: Done uploading piece. err: <nil> Dec 23 14:28:04.655099941 (430h59m4.526182986s) [noise=true , address=145.79.227.38:4978 , piece_id=7M2D3NR342SZVWHS754YCZYVPX2SAKZDUHNV5MIUMG7NQYMJORXA, seg_pos={0 0}, upload=378343]: Done uploading piece. err: <nil> Dec 23 14:28:04.663569098 (430h59m4.534652133s) [noise=true , address=145.79.227.15:4873 , piece_id=RROMZU3GQXWHZ2KAHKJNUXB6JYJW5DOZIMS4RTI2CK4H6GJ4JJPA, seg_pos={0 0}, upload=378343]: Done uploading piece. err: <nil> Dec 23 14:28:04.784039696 (430h59m4.655122741s) [noise=true , address=145.79.227.39:4014 , piece_id=UIYHVARR7QJN7TC45DPH7YMWLOV27EVHFR4HNZXTNGCFCYZ5QWSQ, seg_pos={0 0}, upload=378343]: Done uploading piece. err: <nil> Dec 23 14:28:04.789452723 (430h59m4.660535769s) [noise=true , address=145.79.227.50:4375 , piece_id=2L6LRC33345CFQ776H4C3RBHD27J3WN5BBB7QUEQ43EQED5LQJRA, seg_pos={0 0}, upload=378343]: Done uploading piece. err: <nil> Dec 23 14:28:04.809692453 (430h59m4.680775488s) [noise=true , address=145.79.227.32:4834 , piece_id=FLPDIEXB34RV3VYR4LZYW63M76C3G2352VTJPF6IHDJSKAY2VSIQ, seg_pos={0 0}, upload=378343]: Done uploading piece. err: <nil> Dec 23 14:28:04.814157645 (430h59m4.68524069s) [noise=true , address=145.79.227.20:4438 , piece_id=OHBU6DZOK7H6MIDZHMXEPL2MDY2OXQSX3VPDIE3RDW2THSMC2IMA, seg_pos={0 0}, upload=378343]: Done uploading piece. err: <nil> Dec 23 14:28:04.831851378 (430h59m4.702934423s) [noise=true , address=45.91.116.6:4975 , piece_id=3MBFI2ILY6VP6XT2PMXFVWG7UA2HYOYYNZASSLX3MLQWTEV7EAKA, seg_pos={0 0}, upload=378343]: Done uploading piece. err: <nil> Dec 23 14:28:04.884609870 (430h59m4.755692915s) [noise=true , address=45.91.116.4:4939 , piece_id=3BWKNBSDTQ4DBJFYXFY3IPWOJ3JTPPDMRO2DRYA5REJ3BERVMFKQ, seg_pos={0 0}, upload=378343]: Done uploading piece. err: <nil> Dec 23 14:28:05.131970658 (430h59m5.003053693s) [noise=true , address=77.90.31.14:4664 , piece_id=TGN5OFP66O6EI654N63FVJSCOXFSIS3AURJGAGYWLSBHVMAOFYFQ, seg_pos={0 0}, upload=378343]: Done uploading piece. err: <nil> Dec 23 14:28:05.195398928 (430h59m5.066481983s) [noise=true , address=145.79.227.51:4410 , piece_id=T5XXHOFBLW3A3OKTDY5CTO74VFCY5ZQJRJPACYF3SCLOTF6E25LA, seg_pos={0 0}, upload=378343]: Done uploading piece. err: <nil> Dec 23 14:28:05.218188272 (430h59m5.089271317s) [noise=true , address=145.79.227.11:4731 , piece_id=JDCFIGBBNHW67SQCYI7PKCKWPUQBCGO3BZROXPHNABIGFFMATMOQ, seg_pos={0 0}, upload=378343]: Done uploading piece. err: <nil> Dec 23 14:28:05.218214873 (430h59m5.089297908s) [seg_pos={0 0}, upload=378343]: Segment reached optimal threshold of 13 pieces. Dec 23 14:28:05.218460901 (430h59m5.089543966s) [noise=true , address=145.79.227.37:4565 , piece_id=RIXR6277D5GYIDPGNMMSJAQLFCMM4P6LWALPUSNFW67BWRXQJ4SQ, seg_pos={0 0}, upload=378343]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nvvNczHtQrSEJuJbErnjiAWa4BRDzBrQmBNbB7RrrqN1JWNHT): protocol: write tcp 188.166.225.170:39840->145.79.227.37:4565: use of closed network connection; write tcp 188.166.225.170:39840->145.79.227.37:4565: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39840->145.79.227.37:4565: use of closed network connection Dec 23 14:28:05.218469708 (430h59m5.089552753s) [noise=true , address=145.79.227.37:4565 , piece_id=RIXR6277D5GYIDPGNMMSJAQLFCMM4P6LWALPUSNFW67BWRXQJ4SQ, seg_pos={0 0}, upload=378343]: Long tail cancellation due to optimal threshold Dec 23 14:28:05.218750874 (430h59m5.089833919s) [noise=true , address=145.79.227.40:4049 , piece_id=3QAECV5EQI3BBHUW7AZXVYWZY6DAOOH6DVP4XAMDXBFJAV34QB5Q, seg_pos={0 0}, upload=378343]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF Dec 23 14:28:05.218770391 (430h59m5.089853436s) [noise=true , address=145.79.227.40:4049 , piece_id=3QAECV5EQI3BBHUW7AZXVYWZY6DAOOH6DVP4XAMDXBFJAV34QB5Q, seg_pos={0 0}, upload=378343]: Long tail cancellation due to optimal threshold Dec 23 14:28:05.219134284 (430h59m5.090217349s) [noise=true , address=145.79.227.52:4445 , piece_id=VIE24CGHGMPQSCGUQKOGNIOMJMZHYD4BRZSGEPMUSPPAXPSFOHZQ, seg_pos={0 0}, upload=378343]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12vXrkhLqCofhR4Fmfqb98gxnqWQF6vMV4q3ATbRvPfcqi82WSo): protocol: expected piece hash; context canceled; EOF Dec 23 14:28:05.219146146 (430h59m5.090229201s) [noise=true , address=145.79.227.52:4445 , piece_id=VIE24CGHGMPQSCGUQKOGNIOMJMZHYD4BRZSGEPMUSPPAXPSFOHZQ, seg_pos={0 0}, upload=378343]: Long tail cancellation due to optimal threshold Dec 23 14:28:05.219173128 (430h59m5.090256173s) [seg_pos={0 0}, upload=378343]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 14:28:05.219417493 (430h59m5.090500538s) [upload=378343]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 14:28:05.219425228 (430h59m5.090508293s) [upload=378343]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 14:28:05.522079636 (430h59m5.393162671s) [upload=378343]: Done uploading segments... Dec 23 14:28:05.522100636 (430h59m5.393183681s) [upload=378343]: Upload finished. err: <nil> Duration: 13.307368109s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [.X...............X.....X............X.......................] duration: 3.721339667s trace: https://jaeger.dev.storj.io/trace/772f273c480c0b2b FAIL (2025-12-23 14:30:32 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.721402646s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.......................X..........X.XX..X............XX..X..] duration: 7.844301893s trace: https://jaeger.dev.storj.io/trace/215296e6cb5de778 FAIL (2025-12-23 14:29:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9, Piece ID: TGN5OFP66O6EI654N63FVJSCOXFSIS3AURJGAGYWLSBHVMAOFYFQ) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: 3MBFI2ILY6VP6XT2PMXFVWG7UA2HYOYYNZASSLX3MLQWTEV7EAKA) context deadline exceeded ======LOGS====== Dec 23 14:28:02.333620097 (430h59m2.204703152s) [upload=378343]: Starting upload Dec 23 14:28:02.333737752 (430h59m2.204820797s) [upload=378343]: Done starting upload Dec 23 14:28:02.333838052 (430h59m2.204921107s) [upload=378343]: Uploading segments... Dec 23 14:28:02.334165316 (430h59m2.205248381s) [upload=378343]: Got next segment. Inline: false Dec 23 14:28:02.334217274 (430h59m2.205300319s) [upload=378343]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 14:28:02.334693501 (430h59m2.205776567s) [upload=378343]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 14:28:02.588168613 (430h59m2.459251658s) [seg_pos={0 0}, upload=378343]: Begin upload segment... Dec 23 14:28:02.588224871 (430h59m2.459307916s) [seg_pos={0 0}, upload=378343]: Done begin upload segment. Dec 23 14:28:02.588264035 (430h59m2.45934707s) [noise=true , address=145.79.227.32:4834 , piece_id=FLPDIEXB34RV3VYR4LZYW63M76C3G2352VTJPF6IHDJSKAY2VSIQ, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.588315313 (430h59m2.459398349s) [noise=true , address=145.79.227.11:4731 , piece_id=JDCFIGBBNHW67SQCYI7PKCKWPUQBCGO3BZROXPHNABIGFFMATMOQ, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.589826190 (430h59m2.460909225s) [noise=true , address=145.79.227.37:4565 , piece_id=RIXR6277D5GYIDPGNMMSJAQLFCMM4P6LWALPUSNFW67BWRXQJ4SQ, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.589974662 (430h59m2.461057708s) [noise=true , address=145.79.227.50:4375 , piece_id=2L6LRC33345CFQ776H4C3RBHD27J3WN5BBB7QUEQ43EQED5LQJRA, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.590200293 (430h59m2.461283328s) [noise=true , address=145.79.227.52:4445 , piece_id=VIE24CGHGMPQSCGUQKOGNIOMJMZHYD4BRZSGEPMUSPPAXPSFOHZQ, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.590571109 (430h59m2.461654154s) [noise=true , address=145.79.227.39:4014 , piece_id=UIYHVARR7QJN7TC45DPH7YMWLOV27EVHFR4HNZXTNGCFCYZ5QWSQ, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.590591117 (430h59m2.461674162s) [noise=true , address=77.90.31.14:4664 , piece_id=TGN5OFP66O6EI654N63FVJSCOXFSIS3AURJGAGYWLSBHVMAOFYFQ, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.591120444 (430h59m2.462203489s) [noise=true , address=145.79.227.38:4978 , piece_id=7M2D3NR342SZVWHS754YCZYVPX2SAKZDUHNV5MIUMG7NQYMJORXA, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.591482514 (430h59m2.462565559s) [noise=true , address=145.79.227.15:4873 , piece_id=RROMZU3GQXWHZ2KAHKJNUXB6JYJW5DOZIMS4RTI2CK4H6GJ4JJPA, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.592004899 (430h59m2.463087944s) [noise=true , address=45.91.116.6:4975 , piece_id=3MBFI2ILY6VP6XT2PMXFVWG7UA2HYOYYNZASSLX3MLQWTEV7EAKA, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.592099669 (430h59m2.463182724s) [noise=true , address=145.79.227.20:4438 , piece_id=OHBU6DZOK7H6MIDZHMXEPL2MDY2OXQSX3VPDIE3RDW2THSMC2IMA, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.593944512 (430h59m2.465027557s) [noise=true , address=145.79.227.16:4910 , piece_id=FDXOMBKKQQPPL3G577PWWCWDJFSGZ3W3Q6OZYANVQ7XSAINJ7MFA, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.597912307 (430h59m2.468995352s) [noise=true , address=145.79.227.29:4943 , piece_id=FF4Q6CY7O4EUFQO5QY7GHSYAAQFHK2WCH7A2BR5B6JGJH4MGHSTA, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.600520093 (430h59m2.471603128s) [noise=true , address=145.79.227.40:4049 , piece_id=3QAECV5EQI3BBHUW7AZXVYWZY6DAOOH6DVP4XAMDXBFJAV34QB5Q, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.602548155 (430h59m2.47363121s) [noise=true , address=45.91.116.4:4939 , piece_id=3BWKNBSDTQ4DBJFYXFY3IPWOJ3JTPPDMRO2DRYA5REJ3BERVMFKQ, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:02.604654816 (430h59m2.475737861s) [noise=true , address=145.79.227.51:4410 , piece_id=T5XXHOFBLW3A3OKTDY5CTO74VFCY5ZQJRJPACYF3SCLOTF6E25LA, seg_pos={0 0}, upload=378343]: Uploading piece... Dec 23 14:28:04.341061516 (430h59m4.212144561s) [upload=378343]: Next returned nil segment Dec 23 14:28:04.341149894 (430h59m4.212232949s) [upload=378343]: Waiting for error group managing segments... Dec 23 14:28:04.631523518 (430h59m4.502606563s) [noise=true , address=145.79.227.16:4910 , piece_id=FDXOMBKKQQPPL3G577PWWCWDJFSGZ3W3Q6OZYANVQ7XSAINJ7MFA, seg_pos={0 0}, upload=378343]: Done uploading piece. err: <nil> Dec 23 14:28:04.649224493 (430h59m4.520307538s) [noise=true , address=145.79.227.29:4943 , piece_id=FF4Q6CY7O4EUFQO5QY7GHSYAAQFHK2WCH7A2BR5B6JGJH4MGHSTA, seg_pos={0 0}, upload=378343]: Done uploading piece. err: <nil> Dec 23 14:28:04.655099941 (430h59m4.526182986s) [noise=true , address=145.79.227.38:4978 , piece_id=7M2D3NR342SZVWHS754YCZYVPX2SAKZDUHNV5MIUMG7NQYMJORXA, seg_pos={0 0}, upload=378343]: Done uploading piece. err: <nil> Dec 23 14:28:04.663569098 (430h59m4.534652133s) [noise=true , address=145.79.227.15:4873 , piece_id=RROMZU3GQXWHZ2KAHKJNUXB6JYJW5DOZIMS4RTI2CK4H6GJ4JJPA, seg_pos={0 0}, upload=378343]: Done uploading piece. err: <nil> Dec 23 14:28:04.784039696 (430h59m4.655122741s) [noise=true , address=145.79.227.39:4014 , piece_id=UIYHVARR7QJN7TC45DPH7YMWLOV27EVHFR4HNZXTNGCFCYZ5QWSQ, seg_pos={0 0}, upload=378343]: Done uploading piece. err: <nil> Dec 23 14:28:04.789452723 (430h59m4.660535769s) [noise=true , address=145.79.227.50:4375 , piece_id=2L6LRC33345CFQ776H4C3RBHD27J3WN5BBB7QUEQ43EQED5LQJRA, seg_pos={0 0}, upload=378343]: Done uploading piece. err: <nil> Dec 23 14:28:04.809692453 (430h59m4.680775488s) [noise=true , address=145.79.227.32:4834 , piece_id=FLPDIEXB34RV3VYR4LZYW63M76C3G2352VTJPF6IHDJSKAY2VSIQ, seg_pos={0 0}, upload=378343]: Done uploading piece. err: <nil> Dec 23 14:28:04.814157645 (430h59m4.68524069s) [noise=true , address=145.79.227.20:4438 , piece_id=OHBU6DZOK7H6MIDZHMXEPL2MDY2OXQSX3VPDIE3RDW2THSMC2IMA, seg_pos={0 0}, upload=378343]: Done uploading piece. err: <nil> Dec 23 14:28:04.831851378 (430h59m4.702934423s) [noise=true , address=45.91.116.6:4975 , piece_id=3MBFI2ILY6VP6XT2PMXFVWG7UA2HYOYYNZASSLX3MLQWTEV7EAKA, seg_pos={0 0}, upload=378343]: Done uploading piece. err: <nil> Dec 23 14:28:04.884609870 (430h59m4.755692915s) [noise=true , address=45.91.116.4:4939 , piece_id=3BWKNBSDTQ4DBJFYXFY3IPWOJ3JTPPDMRO2DRYA5REJ3BERVMFKQ, seg_pos={0 0}, upload=378343]: Done uploading piece. err: <nil> Dec 23 14:28:05.131970658 (430h59m5.003053693s) [noise=true , address=77.90.31.14:4664 , piece_id=TGN5OFP66O6EI654N63FVJSCOXFSIS3AURJGAGYWLSBHVMAOFYFQ, seg_pos={0 0}, upload=378343]: Done uploading piece. err: <nil> Dec 23 14:28:05.195398928 (430h59m5.066481983s) [noise=true , address=145.79.227.51:4410 , piece_id=T5XXHOFBLW3A3OKTDY5CTO74VFCY5ZQJRJPACYF3SCLOTF6E25LA, seg_pos={0 0}, upload=378343]: Done uploading piece. err: <nil> Dec 23 14:28:05.218188272 (430h59m5.089271317s) [noise=true , address=145.79.227.11:4731 , piece_id=JDCFIGBBNHW67SQCYI7PKCKWPUQBCGO3BZROXPHNABIGFFMATMOQ, seg_pos={0 0}, upload=378343]: Done uploading piece. err: <nil> Dec 23 14:28:05.218214873 (430h59m5.089297908s) [seg_pos={0 0}, upload=378343]: Segment reached optimal threshold of 13 pieces. Dec 23 14:28:05.218460901 (430h59m5.089543966s) [noise=true , address=145.79.227.37:4565 , piece_id=RIXR6277D5GYIDPGNMMSJAQLFCMM4P6LWALPUSNFW67BWRXQJ4SQ, seg_pos={0 0}, upload=378343]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nvvNczHtQrSEJuJbErnjiAWa4BRDzBrQmBNbB7RrrqN1JWNHT): protocol: write tcp 188.166.225.170:39840->145.79.227.37:4565: use of closed network connection; write tcp 188.166.225.170:39840->145.79.227.37:4565: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39840->145.79.227.37:4565: use of closed network connection Dec 23 14:28:05.218469708 (430h59m5.089552753s) [noise=true , address=145.79.227.37:4565 , piece_id=RIXR6277D5GYIDPGNMMSJAQLFCMM4P6LWALPUSNFW67BWRXQJ4SQ, seg_pos={0 0}, upload=378343]: Long tail cancellation due to optimal threshold Dec 23 14:28:05.218750874 (430h59m5.089833919s) [noise=true , address=145.79.227.40:4049 , piece_id=3QAECV5EQI3BBHUW7AZXVYWZY6DAOOH6DVP4XAMDXBFJAV34QB5Q, seg_pos={0 0}, upload=378343]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF Dec 23 14:28:05.218770391 (430h59m5.089853436s) [noise=true , address=145.79.227.40:4049 , piece_id=3QAECV5EQI3BBHUW7AZXVYWZY6DAOOH6DVP4XAMDXBFJAV34QB5Q, seg_pos={0 0}, upload=378343]: Long tail cancellation due to optimal threshold Dec 23 14:28:05.219134284 (430h59m5.090217349s) [noise=true , address=145.79.227.52:4445 , piece_id=VIE24CGHGMPQSCGUQKOGNIOMJMZHYD4BRZSGEPMUSPPAXPSFOHZQ, seg_pos={0 0}, upload=378343]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12vXrkhLqCofhR4Fmfqb98gxnqWQF6vMV4q3ATbRvPfcqi82WSo): protocol: expected piece hash; context canceled; EOF Dec 23 14:28:05.219146146 (430h59m5.090229201s) [noise=true , address=145.79.227.52:4445 , piece_id=VIE24CGHGMPQSCGUQKOGNIOMJMZHYD4BRZSGEPMUSPPAXPSFOHZQ, seg_pos={0 0}, upload=378343]: Long tail cancellation due to optimal threshold Dec 23 14:28:05.219173128 (430h59m5.090256173s) [seg_pos={0 0}, upload=378343]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 14:28:05.219417493 (430h59m5.090500538s) [upload=378343]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 14:28:05.219425228 (430h59m5.090508293s) [upload=378343]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 14:28:05.522079636 (430h59m5.393162671s) [upload=378343]: Done uploading segments... Dec 23 14:28:05.522100636 (430h59m5.393183681s) [upload=378343]: Upload finished. err: <nil> Duration: 7.844331841s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)