Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 5.808622438s trace: https://jaeger.dev.storj.io/trace/681cf842d2260d41 FAIL (2025-12-25 05:02:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, Piece ID: ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ) context deadline exceeded; piecestore: (Node ID: 1DT68m6J4PpDk3XUDhDaaqV46NLKVz4A23716JLFxEAh122GeR, Piece ID: WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA) context deadline exceeded; piecestore: (Node ID: 12wWYnhWHSqqAFgje4p9Cr7WYaJ6dJWsCqvrx8ZqVFoakjuFeka, Piece ID: GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q) context deadline exceeded ======LOGS====== nline: false Dec 25 04:40:39.039085560 (469h11m38.910168605s) [upload=411834]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 04:40:39.039095028 (469h11m38.910178073s) [upload=411834]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 04:40:39.297407613 (469h11m39.168490658s) [seg_pos={0 0}, upload=411834]: Begin upload segment... Dec 25 04:40:39.297471084 (469h11m39.168554119s) [seg_pos={0 0}, upload=411834]: Done begin upload segment. Dec 25 04:40:39.297511581 (469h11m39.168594616s) [noise=true , address=145.79.227.51:4411 , piece_id=WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.297663310 (469h11m39.168746345s) [noise=true , address=145.79.227.17:4970 , piece_id=6DPKJ762IUN4A7UEZXLDERCVDW24QEKGEHFETAYZOKWQPVN2QJYA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.298405574 (469h11m39.169488619s) [noise=true , address=145.79.227.2:4114 , piece_id=5ISXZKQSH4CLLTF3ILGAXHV6STMLRUSZP6UATU2677X2FUPLBCRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.298797961 (469h11m39.169881006s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299124392 (469h11m39.170207437s) [noise=true , address=45.91.116.5:4300 , piece_id=NTWC3RWYQQBBKPMXGUG44R2B5GWC2Z4LDFU3GTWOWCT3JIRIZOOQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299438650 (469h11m39.170521685s) [noise=true , address=145.79.227.5:4221 , piece_id=C4SXUVTG2TZ4EIRBGL4TLOLBMYOAZ4KLIQSZOXKEOWAJBQQJLWSA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299794709 (469h11m39.170877744s) [noise=true , address=145.79.227.37:4567 , piece_id=TZ4C57ISUI3CB3VOLTSM5W5QVDI7PVXV67PFJI3ZSVSXHTKPYPSA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.300142221 (469h11m39.171225256s) [noise=true , address=145.79.227.8:4331 , piece_id=ELLVNJFOVY5HSAHZDQYGQVO7OYHFZNFHCBJKHQFV5K6EV2M3JTRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.300506354 (469h11m39.171589389s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.301142795 (469h11m39.17222584s) [noise=true , address=145.79.227.36:4517 , piece_id=HBQU7CMAO2KJS6ODW2WHICBAWQSH6TQQPOQSENKDEISWCWFN2CMA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.301625835 (469h11m39.17270891s) [noise=true , address=145.79.227.45:4229 , piece_id=GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.302170964 (469h11m39.173254009s) [noise=true , address=145.79.227.3:4150 , piece_id=OHGA4EA33AVEDCJLB6QSUJLD3W2TV6YNN6FBVFFKTTSF2B6XDPRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.302629947 (469h11m39.173712982s) [noise=true , address=145.79.227.19:4079 , piece_id=ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303079704 (469h11m39.174162749s) [noise=true , address=145.79.227.15:4875 , piece_id=FUMLT2ODHUE3ETSNWVKBYLSJXYVCHUW5KAJJE5AECNL3IAQYYCLQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303499804 (469h11m39.174582849s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303921386 (469h11m39.175004432s) [noise=true , address=145.79.227.47:4265 , piece_id=TVUXNR6K35A3CWFDN4JRKM4Q6RGBC4LP2XKPXJFN2AJL32ESEBPA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:41.172453460 (469h11m41.043536495s) [upload=411834]: Next returned nil segment Dec 25 04:40:41.172473798 (469h11m41.043556833s) [upload=411834]: Waiting for error group managing segments... Dec 25 04:40:41.483361556 (469h11m41.354444601s) [noise=true , address=145.79.227.37:4567 , piece_id=TZ4C57ISUI3CB3VOLTSM5W5QVDI7PVXV67PFJI3ZSVSXHTKPYPSA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.484283170 (469h11m41.355366205s) [noise=true , address=145.79.227.5:4221 , piece_id=C4SXUVTG2TZ4EIRBGL4TLOLBMYOAZ4KLIQSZOXKEOWAJBQQJLWSA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.515393699 (469h11m41.386476744s) [noise=true , address=145.79.227.51:4411 , piece_id=WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.526179387 (469h11m41.397262422s) [noise=true , address=45.91.116.5:4300 , piece_id=NTWC3RWYQQBBKPMXGUG44R2B5GWC2Z4LDFU3GTWOWCT3JIRIZOOQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.642380892 (469h11m41.513463937s) [noise=true , address=145.79.227.36:4517 , piece_id=HBQU7CMAO2KJS6ODW2WHICBAWQSH6TQQPOQSENKDEISWCWFN2CMA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.662326662 (469h11m41.533409707s) [noise=true , address=145.79.227.3:4150 , piece_id=OHGA4EA33AVEDCJLB6QSUJLD3W2TV6YNN6FBVFFKTTSF2B6XDPRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.758852130 (469h11m41.629935175s) [noise=true , address=145.79.227.47:4265 , piece_id=TVUXNR6K35A3CWFDN4JRKM4Q6RGBC4LP2XKPXJFN2AJL32ESEBPA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.823449082 (469h11m41.694532117s) [noise=true , address=145.79.227.17:4970 , piece_id=6DPKJ762IUN4A7UEZXLDERCVDW24QEKGEHFETAYZOKWQPVN2QJYA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.952636616 (469h11m41.823719661s) [noise=true , address=145.79.227.19:4079 , piece_id=ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.025177123 (469h11m41.896260168s) [noise=true , address=145.79.227.45:4229 , piece_id=GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.222526274 (469h11m42.093609319s) [noise=true , address=145.79.227.8:4331 , piece_id=ELLVNJFOVY5HSAHZDQYGQVO7OYHFZNFHCBJKHQFV5K6EV2M3JTRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.250024374 (469h11m42.121107409s) [noise=true , address=145.79.227.2:4114 , piece_id=5ISXZKQSH4CLLTF3ILGAXHV6STMLRUSZP6UATU2677X2FUPLBCRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.651787921 (469h11m42.522870966s) [noise=true , address=145.79.227.15:4875 , piece_id=FUMLT2ODHUE3ETSNWVKBYLSJXYVCHUW5KAJJE5AECNL3IAQYYCLQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.651814982 (469h11m42.522898027s) [seg_pos={0 0}, upload=411834]: Segment reached optimal threshold of 13 pieces. Dec 25 04:40:42.652052093 (469h11m42.523135129s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: write tcp 188.166.225.170:36650->77.90.31.15:4697: use of closed network connection; storage node overloaded, request limit: 2000 Dec 25 04:40:42.652069537 (469h11m42.523152572s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652177673 (469h11m42.523260708s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd): protocol: write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection; write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection Dec 25 04:40:42.652184456 (469h11m42.523267501s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652230243 (469h11m42.523313308s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq): protocol: write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection Dec 25 04:40:42.652240693 (469h11m42.523323778s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652268265 (469h11m42.52335131s) [seg_pos={0 0}, upload=411834]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 04:40:42.652354770 (469h11m42.523437815s) [upload=411834]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 04:40:42.652361693 (469h11m42.523444738s) [upload=411834]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 04:40:42.959704440 (469h11m42.830787485s) [upload=411834]: Done uploading segments... Dec 25 04:40:42.959728234 (469h11m42.83081127s) [upload=411834]: Upload finished. err: <nil> Duration: 5.808651184s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 6.01305729s trace: https://jaeger.dev.storj.io/trace/681cf842d2260d41 FAIL (2025-12-25 05:01:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, Piece ID: ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ) context deadline exceeded; piecestore: (Node ID: 1DT68m6J4PpDk3XUDhDaaqV46NLKVz4A23716JLFxEAh122GeR, Piece ID: WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA) context deadline exceeded; piecestore: (Node ID: 12wWYnhWHSqqAFgje4p9Cr7WYaJ6dJWsCqvrx8ZqVFoakjuFeka, Piece ID: GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q) context deadline exceeded ======LOGS====== nline: false Dec 25 04:40:39.039085560 (469h11m38.910168605s) [upload=411834]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 04:40:39.039095028 (469h11m38.910178073s) [upload=411834]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 04:40:39.297407613 (469h11m39.168490658s) [seg_pos={0 0}, upload=411834]: Begin upload segment... Dec 25 04:40:39.297471084 (469h11m39.168554119s) [seg_pos={0 0}, upload=411834]: Done begin upload segment. Dec 25 04:40:39.297511581 (469h11m39.168594616s) [noise=true , address=145.79.227.51:4411 , piece_id=WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.297663310 (469h11m39.168746345s) [noise=true , address=145.79.227.17:4970 , piece_id=6DPKJ762IUN4A7UEZXLDERCVDW24QEKGEHFETAYZOKWQPVN2QJYA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.298405574 (469h11m39.169488619s) [noise=true , address=145.79.227.2:4114 , piece_id=5ISXZKQSH4CLLTF3ILGAXHV6STMLRUSZP6UATU2677X2FUPLBCRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.298797961 (469h11m39.169881006s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299124392 (469h11m39.170207437s) [noise=true , address=45.91.116.5:4300 , piece_id=NTWC3RWYQQBBKPMXGUG44R2B5GWC2Z4LDFU3GTWOWCT3JIRIZOOQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299438650 (469h11m39.170521685s) [noise=true , address=145.79.227.5:4221 , piece_id=C4SXUVTG2TZ4EIRBGL4TLOLBMYOAZ4KLIQSZOXKEOWAJBQQJLWSA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299794709 (469h11m39.170877744s) [noise=true , address=145.79.227.37:4567 , piece_id=TZ4C57ISUI3CB3VOLTSM5W5QVDI7PVXV67PFJI3ZSVSXHTKPYPSA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.300142221 (469h11m39.171225256s) [noise=true , address=145.79.227.8:4331 , piece_id=ELLVNJFOVY5HSAHZDQYGQVO7OYHFZNFHCBJKHQFV5K6EV2M3JTRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.300506354 (469h11m39.171589389s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.301142795 (469h11m39.17222584s) [noise=true , address=145.79.227.36:4517 , piece_id=HBQU7CMAO2KJS6ODW2WHICBAWQSH6TQQPOQSENKDEISWCWFN2CMA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.301625835 (469h11m39.17270891s) [noise=true , address=145.79.227.45:4229 , piece_id=GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.302170964 (469h11m39.173254009s) [noise=true , address=145.79.227.3:4150 , piece_id=OHGA4EA33AVEDCJLB6QSUJLD3W2TV6YNN6FBVFFKTTSF2B6XDPRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.302629947 (469h11m39.173712982s) [noise=true , address=145.79.227.19:4079 , piece_id=ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303079704 (469h11m39.174162749s) [noise=true , address=145.79.227.15:4875 , piece_id=FUMLT2ODHUE3ETSNWVKBYLSJXYVCHUW5KAJJE5AECNL3IAQYYCLQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303499804 (469h11m39.174582849s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303921386 (469h11m39.175004432s) [noise=true , address=145.79.227.47:4265 , piece_id=TVUXNR6K35A3CWFDN4JRKM4Q6RGBC4LP2XKPXJFN2AJL32ESEBPA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:41.172453460 (469h11m41.043536495s) [upload=411834]: Next returned nil segment Dec 25 04:40:41.172473798 (469h11m41.043556833s) [upload=411834]: Waiting for error group managing segments... Dec 25 04:40:41.483361556 (469h11m41.354444601s) [noise=true , address=145.79.227.37:4567 , piece_id=TZ4C57ISUI3CB3VOLTSM5W5QVDI7PVXV67PFJI3ZSVSXHTKPYPSA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.484283170 (469h11m41.355366205s) [noise=true , address=145.79.227.5:4221 , piece_id=C4SXUVTG2TZ4EIRBGL4TLOLBMYOAZ4KLIQSZOXKEOWAJBQQJLWSA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.515393699 (469h11m41.386476744s) [noise=true , address=145.79.227.51:4411 , piece_id=WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.526179387 (469h11m41.397262422s) [noise=true , address=45.91.116.5:4300 , piece_id=NTWC3RWYQQBBKPMXGUG44R2B5GWC2Z4LDFU3GTWOWCT3JIRIZOOQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.642380892 (469h11m41.513463937s) [noise=true , address=145.79.227.36:4517 , piece_id=HBQU7CMAO2KJS6ODW2WHICBAWQSH6TQQPOQSENKDEISWCWFN2CMA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.662326662 (469h11m41.533409707s) [noise=true , address=145.79.227.3:4150 , piece_id=OHGA4EA33AVEDCJLB6QSUJLD3W2TV6YNN6FBVFFKTTSF2B6XDPRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.758852130 (469h11m41.629935175s) [noise=true , address=145.79.227.47:4265 , piece_id=TVUXNR6K35A3CWFDN4JRKM4Q6RGBC4LP2XKPXJFN2AJL32ESEBPA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.823449082 (469h11m41.694532117s) [noise=true , address=145.79.227.17:4970 , piece_id=6DPKJ762IUN4A7UEZXLDERCVDW24QEKGEHFETAYZOKWQPVN2QJYA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.952636616 (469h11m41.823719661s) [noise=true , address=145.79.227.19:4079 , piece_id=ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.025177123 (469h11m41.896260168s) [noise=true , address=145.79.227.45:4229 , piece_id=GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.222526274 (469h11m42.093609319s) [noise=true , address=145.79.227.8:4331 , piece_id=ELLVNJFOVY5HSAHZDQYGQVO7OYHFZNFHCBJKHQFV5K6EV2M3JTRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.250024374 (469h11m42.121107409s) [noise=true , address=145.79.227.2:4114 , piece_id=5ISXZKQSH4CLLTF3ILGAXHV6STMLRUSZP6UATU2677X2FUPLBCRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.651787921 (469h11m42.522870966s) [noise=true , address=145.79.227.15:4875 , piece_id=FUMLT2ODHUE3ETSNWVKBYLSJXYVCHUW5KAJJE5AECNL3IAQYYCLQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.651814982 (469h11m42.522898027s) [seg_pos={0 0}, upload=411834]: Segment reached optimal threshold of 13 pieces. Dec 25 04:40:42.652052093 (469h11m42.523135129s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: write tcp 188.166.225.170:36650->77.90.31.15:4697: use of closed network connection; storage node overloaded, request limit: 2000 Dec 25 04:40:42.652069537 (469h11m42.523152572s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652177673 (469h11m42.523260708s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd): protocol: write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection; write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection Dec 25 04:40:42.652184456 (469h11m42.523267501s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652230243 (469h11m42.523313308s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq): protocol: write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection Dec 25 04:40:42.652240693 (469h11m42.523323778s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652268265 (469h11m42.52335131s) [seg_pos={0 0}, upload=411834]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 04:40:42.652354770 (469h11m42.523437815s) [upload=411834]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 04:40:42.652361693 (469h11m42.523444738s) [upload=411834]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 04:40:42.959704440 (469h11m42.830787485s) [upload=411834]: Done uploading segments... Dec 25 04:40:42.959728234 (469h11m42.83081127s) [upload=411834]: Upload finished. err: <nil> Duration: 6.01309418s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 6.122129603s trace: https://jaeger.dev.storj.io/trace/681cf842d2260d41 FAIL (2025-12-25 05:01:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, Piece ID: ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ) context deadline exceeded; piecestore: (Node ID: 1DT68m6J4PpDk3XUDhDaaqV46NLKVz4A23716JLFxEAh122GeR, Piece ID: WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA) context deadline exceeded; piecestore: (Node ID: 12wWYnhWHSqqAFgje4p9Cr7WYaJ6dJWsCqvrx8ZqVFoakjuFeka, Piece ID: GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q) context deadline exceeded ======LOGS====== nline: false Dec 25 04:40:39.039085560 (469h11m38.910168605s) [upload=411834]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 04:40:39.039095028 (469h11m38.910178073s) [upload=411834]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 04:40:39.297407613 (469h11m39.168490658s) [seg_pos={0 0}, upload=411834]: Begin upload segment... Dec 25 04:40:39.297471084 (469h11m39.168554119s) [seg_pos={0 0}, upload=411834]: Done begin upload segment. Dec 25 04:40:39.297511581 (469h11m39.168594616s) [noise=true , address=145.79.227.51:4411 , piece_id=WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.297663310 (469h11m39.168746345s) [noise=true , address=145.79.227.17:4970 , piece_id=6DPKJ762IUN4A7UEZXLDERCVDW24QEKGEHFETAYZOKWQPVN2QJYA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.298405574 (469h11m39.169488619s) [noise=true , address=145.79.227.2:4114 , piece_id=5ISXZKQSH4CLLTF3ILGAXHV6STMLRUSZP6UATU2677X2FUPLBCRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.298797961 (469h11m39.169881006s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299124392 (469h11m39.170207437s) [noise=true , address=45.91.116.5:4300 , piece_id=NTWC3RWYQQBBKPMXGUG44R2B5GWC2Z4LDFU3GTWOWCT3JIRIZOOQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299438650 (469h11m39.170521685s) [noise=true , address=145.79.227.5:4221 , piece_id=C4SXUVTG2TZ4EIRBGL4TLOLBMYOAZ4KLIQSZOXKEOWAJBQQJLWSA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299794709 (469h11m39.170877744s) [noise=true , address=145.79.227.37:4567 , piece_id=TZ4C57ISUI3CB3VOLTSM5W5QVDI7PVXV67PFJI3ZSVSXHTKPYPSA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.300142221 (469h11m39.171225256s) [noise=true , address=145.79.227.8:4331 , piece_id=ELLVNJFOVY5HSAHZDQYGQVO7OYHFZNFHCBJKHQFV5K6EV2M3JTRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.300506354 (469h11m39.171589389s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.301142795 (469h11m39.17222584s) [noise=true , address=145.79.227.36:4517 , piece_id=HBQU7CMAO2KJS6ODW2WHICBAWQSH6TQQPOQSENKDEISWCWFN2CMA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.301625835 (469h11m39.17270891s) [noise=true , address=145.79.227.45:4229 , piece_id=GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.302170964 (469h11m39.173254009s) [noise=true , address=145.79.227.3:4150 , piece_id=OHGA4EA33AVEDCJLB6QSUJLD3W2TV6YNN6FBVFFKTTSF2B6XDPRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.302629947 (469h11m39.173712982s) [noise=true , address=145.79.227.19:4079 , piece_id=ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303079704 (469h11m39.174162749s) [noise=true , address=145.79.227.15:4875 , piece_id=FUMLT2ODHUE3ETSNWVKBYLSJXYVCHUW5KAJJE5AECNL3IAQYYCLQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303499804 (469h11m39.174582849s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303921386 (469h11m39.175004432s) [noise=true , address=145.79.227.47:4265 , piece_id=TVUXNR6K35A3CWFDN4JRKM4Q6RGBC4LP2XKPXJFN2AJL32ESEBPA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:41.172453460 (469h11m41.043536495s) [upload=411834]: Next returned nil segment Dec 25 04:40:41.172473798 (469h11m41.043556833s) [upload=411834]: Waiting for error group managing segments... Dec 25 04:40:41.483361556 (469h11m41.354444601s) [noise=true , address=145.79.227.37:4567 , piece_id=TZ4C57ISUI3CB3VOLTSM5W5QVDI7PVXV67PFJI3ZSVSXHTKPYPSA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.484283170 (469h11m41.355366205s) [noise=true , address=145.79.227.5:4221 , piece_id=C4SXUVTG2TZ4EIRBGL4TLOLBMYOAZ4KLIQSZOXKEOWAJBQQJLWSA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.515393699 (469h11m41.386476744s) [noise=true , address=145.79.227.51:4411 , piece_id=WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.526179387 (469h11m41.397262422s) [noise=true , address=45.91.116.5:4300 , piece_id=NTWC3RWYQQBBKPMXGUG44R2B5GWC2Z4LDFU3GTWOWCT3JIRIZOOQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.642380892 (469h11m41.513463937s) [noise=true , address=145.79.227.36:4517 , piece_id=HBQU7CMAO2KJS6ODW2WHICBAWQSH6TQQPOQSENKDEISWCWFN2CMA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.662326662 (469h11m41.533409707s) [noise=true , address=145.79.227.3:4150 , piece_id=OHGA4EA33AVEDCJLB6QSUJLD3W2TV6YNN6FBVFFKTTSF2B6XDPRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.758852130 (469h11m41.629935175s) [noise=true , address=145.79.227.47:4265 , piece_id=TVUXNR6K35A3CWFDN4JRKM4Q6RGBC4LP2XKPXJFN2AJL32ESEBPA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.823449082 (469h11m41.694532117s) [noise=true , address=145.79.227.17:4970 , piece_id=6DPKJ762IUN4A7UEZXLDERCVDW24QEKGEHFETAYZOKWQPVN2QJYA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.952636616 (469h11m41.823719661s) [noise=true , address=145.79.227.19:4079 , piece_id=ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.025177123 (469h11m41.896260168s) [noise=true , address=145.79.227.45:4229 , piece_id=GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.222526274 (469h11m42.093609319s) [noise=true , address=145.79.227.8:4331 , piece_id=ELLVNJFOVY5HSAHZDQYGQVO7OYHFZNFHCBJKHQFV5K6EV2M3JTRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.250024374 (469h11m42.121107409s) [noise=true , address=145.79.227.2:4114 , piece_id=5ISXZKQSH4CLLTF3ILGAXHV6STMLRUSZP6UATU2677X2FUPLBCRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.651787921 (469h11m42.522870966s) [noise=true , address=145.79.227.15:4875 , piece_id=FUMLT2ODHUE3ETSNWVKBYLSJXYVCHUW5KAJJE5AECNL3IAQYYCLQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.651814982 (469h11m42.522898027s) [seg_pos={0 0}, upload=411834]: Segment reached optimal threshold of 13 pieces. Dec 25 04:40:42.652052093 (469h11m42.523135129s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: write tcp 188.166.225.170:36650->77.90.31.15:4697: use of closed network connection; storage node overloaded, request limit: 2000 Dec 25 04:40:42.652069537 (469h11m42.523152572s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652177673 (469h11m42.523260708s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd): protocol: write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection; write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection Dec 25 04:40:42.652184456 (469h11m42.523267501s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652230243 (469h11m42.523313308s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq): protocol: write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection Dec 25 04:40:42.652240693 (469h11m42.523323778s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652268265 (469h11m42.52335131s) [seg_pos={0 0}, upload=411834]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 04:40:42.652354770 (469h11m42.523437815s) [upload=411834]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 04:40:42.652361693 (469h11m42.523444738s) [upload=411834]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 04:40:42.959704440 (469h11m42.830787485s) [upload=411834]: Done uploading segments... Dec 25 04:40:42.959728234 (469h11m42.83081127s) [upload=411834]: Upload finished. err: <nil> Duration: 6.12216021s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 5.921992453s trace: https://jaeger.dev.storj.io/trace/681cf842d2260d41 FAIL (2025-12-25 05:00:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, Piece ID: ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ) context deadline exceeded; piecestore: (Node ID: 1DT68m6J4PpDk3XUDhDaaqV46NLKVz4A23716JLFxEAh122GeR, Piece ID: WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA) context deadline exceeded; piecestore: (Node ID: 12wWYnhWHSqqAFgje4p9Cr7WYaJ6dJWsCqvrx8ZqVFoakjuFeka, Piece ID: GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q) context deadline exceeded ======LOGS====== nline: false Dec 25 04:40:39.039085560 (469h11m38.910168605s) [upload=411834]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 04:40:39.039095028 (469h11m38.910178073s) [upload=411834]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 04:40:39.297407613 (469h11m39.168490658s) [seg_pos={0 0}, upload=411834]: Begin upload segment... Dec 25 04:40:39.297471084 (469h11m39.168554119s) [seg_pos={0 0}, upload=411834]: Done begin upload segment. Dec 25 04:40:39.297511581 (469h11m39.168594616s) [noise=true , address=145.79.227.51:4411 , piece_id=WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.297663310 (469h11m39.168746345s) [noise=true , address=145.79.227.17:4970 , piece_id=6DPKJ762IUN4A7UEZXLDERCVDW24QEKGEHFETAYZOKWQPVN2QJYA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.298405574 (469h11m39.169488619s) [noise=true , address=145.79.227.2:4114 , piece_id=5ISXZKQSH4CLLTF3ILGAXHV6STMLRUSZP6UATU2677X2FUPLBCRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.298797961 (469h11m39.169881006s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299124392 (469h11m39.170207437s) [noise=true , address=45.91.116.5:4300 , piece_id=NTWC3RWYQQBBKPMXGUG44R2B5GWC2Z4LDFU3GTWOWCT3JIRIZOOQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299438650 (469h11m39.170521685s) [noise=true , address=145.79.227.5:4221 , piece_id=C4SXUVTG2TZ4EIRBGL4TLOLBMYOAZ4KLIQSZOXKEOWAJBQQJLWSA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299794709 (469h11m39.170877744s) [noise=true , address=145.79.227.37:4567 , piece_id=TZ4C57ISUI3CB3VOLTSM5W5QVDI7PVXV67PFJI3ZSVSXHTKPYPSA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.300142221 (469h11m39.171225256s) [noise=true , address=145.79.227.8:4331 , piece_id=ELLVNJFOVY5HSAHZDQYGQVO7OYHFZNFHCBJKHQFV5K6EV2M3JTRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.300506354 (469h11m39.171589389s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.301142795 (469h11m39.17222584s) [noise=true , address=145.79.227.36:4517 , piece_id=HBQU7CMAO2KJS6ODW2WHICBAWQSH6TQQPOQSENKDEISWCWFN2CMA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.301625835 (469h11m39.17270891s) [noise=true , address=145.79.227.45:4229 , piece_id=GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.302170964 (469h11m39.173254009s) [noise=true , address=145.79.227.3:4150 , piece_id=OHGA4EA33AVEDCJLB6QSUJLD3W2TV6YNN6FBVFFKTTSF2B6XDPRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.302629947 (469h11m39.173712982s) [noise=true , address=145.79.227.19:4079 , piece_id=ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303079704 (469h11m39.174162749s) [noise=true , address=145.79.227.15:4875 , piece_id=FUMLT2ODHUE3ETSNWVKBYLSJXYVCHUW5KAJJE5AECNL3IAQYYCLQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303499804 (469h11m39.174582849s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303921386 (469h11m39.175004432s) [noise=true , address=145.79.227.47:4265 , piece_id=TVUXNR6K35A3CWFDN4JRKM4Q6RGBC4LP2XKPXJFN2AJL32ESEBPA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:41.172453460 (469h11m41.043536495s) [upload=411834]: Next returned nil segment Dec 25 04:40:41.172473798 (469h11m41.043556833s) [upload=411834]: Waiting for error group managing segments... Dec 25 04:40:41.483361556 (469h11m41.354444601s) [noise=true , address=145.79.227.37:4567 , piece_id=TZ4C57ISUI3CB3VOLTSM5W5QVDI7PVXV67PFJI3ZSVSXHTKPYPSA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.484283170 (469h11m41.355366205s) [noise=true , address=145.79.227.5:4221 , piece_id=C4SXUVTG2TZ4EIRBGL4TLOLBMYOAZ4KLIQSZOXKEOWAJBQQJLWSA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.515393699 (469h11m41.386476744s) [noise=true , address=145.79.227.51:4411 , piece_id=WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.526179387 (469h11m41.397262422s) [noise=true , address=45.91.116.5:4300 , piece_id=NTWC3RWYQQBBKPMXGUG44R2B5GWC2Z4LDFU3GTWOWCT3JIRIZOOQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.642380892 (469h11m41.513463937s) [noise=true , address=145.79.227.36:4517 , piece_id=HBQU7CMAO2KJS6ODW2WHICBAWQSH6TQQPOQSENKDEISWCWFN2CMA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.662326662 (469h11m41.533409707s) [noise=true , address=145.79.227.3:4150 , piece_id=OHGA4EA33AVEDCJLB6QSUJLD3W2TV6YNN6FBVFFKTTSF2B6XDPRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.758852130 (469h11m41.629935175s) [noise=true , address=145.79.227.47:4265 , piece_id=TVUXNR6K35A3CWFDN4JRKM4Q6RGBC4LP2XKPXJFN2AJL32ESEBPA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.823449082 (469h11m41.694532117s) [noise=true , address=145.79.227.17:4970 , piece_id=6DPKJ762IUN4A7UEZXLDERCVDW24QEKGEHFETAYZOKWQPVN2QJYA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.952636616 (469h11m41.823719661s) [noise=true , address=145.79.227.19:4079 , piece_id=ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.025177123 (469h11m41.896260168s) [noise=true , address=145.79.227.45:4229 , piece_id=GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.222526274 (469h11m42.093609319s) [noise=true , address=145.79.227.8:4331 , piece_id=ELLVNJFOVY5HSAHZDQYGQVO7OYHFZNFHCBJKHQFV5K6EV2M3JTRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.250024374 (469h11m42.121107409s) [noise=true , address=145.79.227.2:4114 , piece_id=5ISXZKQSH4CLLTF3ILGAXHV6STMLRUSZP6UATU2677X2FUPLBCRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.651787921 (469h11m42.522870966s) [noise=true , address=145.79.227.15:4875 , piece_id=FUMLT2ODHUE3ETSNWVKBYLSJXYVCHUW5KAJJE5AECNL3IAQYYCLQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.651814982 (469h11m42.522898027s) [seg_pos={0 0}, upload=411834]: Segment reached optimal threshold of 13 pieces. Dec 25 04:40:42.652052093 (469h11m42.523135129s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: write tcp 188.166.225.170:36650->77.90.31.15:4697: use of closed network connection; storage node overloaded, request limit: 2000 Dec 25 04:40:42.652069537 (469h11m42.523152572s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652177673 (469h11m42.523260708s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd): protocol: write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection; write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection Dec 25 04:40:42.652184456 (469h11m42.523267501s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652230243 (469h11m42.523313308s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq): protocol: write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection Dec 25 04:40:42.652240693 (469h11m42.523323778s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652268265 (469h11m42.52335131s) [seg_pos={0 0}, upload=411834]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 04:40:42.652354770 (469h11m42.523437815s) [upload=411834]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 04:40:42.652361693 (469h11m42.523444738s) [upload=411834]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 04:40:42.959704440 (469h11m42.830787485s) [upload=411834]: Done uploading segments... Dec 25 04:40:42.959728234 (469h11m42.83081127s) [upload=411834]: Upload finished. err: <nil> Duration: 5.922025615s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 5.876606178s trace: https://jaeger.dev.storj.io/trace/681cf842d2260d41 FAIL (2025-12-25 04:59:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, Piece ID: ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ) context deadline exceeded; piecestore: (Node ID: 1DT68m6J4PpDk3XUDhDaaqV46NLKVz4A23716JLFxEAh122GeR, Piece ID: WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA) context deadline exceeded; piecestore: (Node ID: 12wWYnhWHSqqAFgje4p9Cr7WYaJ6dJWsCqvrx8ZqVFoakjuFeka, Piece ID: GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q) context deadline exceeded ======LOGS====== nline: false Dec 25 04:40:39.039085560 (469h11m38.910168605s) [upload=411834]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 04:40:39.039095028 (469h11m38.910178073s) [upload=411834]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 04:40:39.297407613 (469h11m39.168490658s) [seg_pos={0 0}, upload=411834]: Begin upload segment... Dec 25 04:40:39.297471084 (469h11m39.168554119s) [seg_pos={0 0}, upload=411834]: Done begin upload segment. Dec 25 04:40:39.297511581 (469h11m39.168594616s) [noise=true , address=145.79.227.51:4411 , piece_id=WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.297663310 (469h11m39.168746345s) [noise=true , address=145.79.227.17:4970 , piece_id=6DPKJ762IUN4A7UEZXLDERCVDW24QEKGEHFETAYZOKWQPVN2QJYA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.298405574 (469h11m39.169488619s) [noise=true , address=145.79.227.2:4114 , piece_id=5ISXZKQSH4CLLTF3ILGAXHV6STMLRUSZP6UATU2677X2FUPLBCRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.298797961 (469h11m39.169881006s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299124392 (469h11m39.170207437s) [noise=true , address=45.91.116.5:4300 , piece_id=NTWC3RWYQQBBKPMXGUG44R2B5GWC2Z4LDFU3GTWOWCT3JIRIZOOQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299438650 (469h11m39.170521685s) [noise=true , address=145.79.227.5:4221 , piece_id=C4SXUVTG2TZ4EIRBGL4TLOLBMYOAZ4KLIQSZOXKEOWAJBQQJLWSA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299794709 (469h11m39.170877744s) [noise=true , address=145.79.227.37:4567 , piece_id=TZ4C57ISUI3CB3VOLTSM5W5QVDI7PVXV67PFJI3ZSVSXHTKPYPSA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.300142221 (469h11m39.171225256s) [noise=true , address=145.79.227.8:4331 , piece_id=ELLVNJFOVY5HSAHZDQYGQVO7OYHFZNFHCBJKHQFV5K6EV2M3JTRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.300506354 (469h11m39.171589389s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.301142795 (469h11m39.17222584s) [noise=true , address=145.79.227.36:4517 , piece_id=HBQU7CMAO2KJS6ODW2WHICBAWQSH6TQQPOQSENKDEISWCWFN2CMA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.301625835 (469h11m39.17270891s) [noise=true , address=145.79.227.45:4229 , piece_id=GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.302170964 (469h11m39.173254009s) [noise=true , address=145.79.227.3:4150 , piece_id=OHGA4EA33AVEDCJLB6QSUJLD3W2TV6YNN6FBVFFKTTSF2B6XDPRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.302629947 (469h11m39.173712982s) [noise=true , address=145.79.227.19:4079 , piece_id=ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303079704 (469h11m39.174162749s) [noise=true , address=145.79.227.15:4875 , piece_id=FUMLT2ODHUE3ETSNWVKBYLSJXYVCHUW5KAJJE5AECNL3IAQYYCLQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303499804 (469h11m39.174582849s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303921386 (469h11m39.175004432s) [noise=true , address=145.79.227.47:4265 , piece_id=TVUXNR6K35A3CWFDN4JRKM4Q6RGBC4LP2XKPXJFN2AJL32ESEBPA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:41.172453460 (469h11m41.043536495s) [upload=411834]: Next returned nil segment Dec 25 04:40:41.172473798 (469h11m41.043556833s) [upload=411834]: Waiting for error group managing segments... Dec 25 04:40:41.483361556 (469h11m41.354444601s) [noise=true , address=145.79.227.37:4567 , piece_id=TZ4C57ISUI3CB3VOLTSM5W5QVDI7PVXV67PFJI3ZSVSXHTKPYPSA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.484283170 (469h11m41.355366205s) [noise=true , address=145.79.227.5:4221 , piece_id=C4SXUVTG2TZ4EIRBGL4TLOLBMYOAZ4KLIQSZOXKEOWAJBQQJLWSA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.515393699 (469h11m41.386476744s) [noise=true , address=145.79.227.51:4411 , piece_id=WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.526179387 (469h11m41.397262422s) [noise=true , address=45.91.116.5:4300 , piece_id=NTWC3RWYQQBBKPMXGUG44R2B5GWC2Z4LDFU3GTWOWCT3JIRIZOOQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.642380892 (469h11m41.513463937s) [noise=true , address=145.79.227.36:4517 , piece_id=HBQU7CMAO2KJS6ODW2WHICBAWQSH6TQQPOQSENKDEISWCWFN2CMA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.662326662 (469h11m41.533409707s) [noise=true , address=145.79.227.3:4150 , piece_id=OHGA4EA33AVEDCJLB6QSUJLD3W2TV6YNN6FBVFFKTTSF2B6XDPRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.758852130 (469h11m41.629935175s) [noise=true , address=145.79.227.47:4265 , piece_id=TVUXNR6K35A3CWFDN4JRKM4Q6RGBC4LP2XKPXJFN2AJL32ESEBPA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.823449082 (469h11m41.694532117s) [noise=true , address=145.79.227.17:4970 , piece_id=6DPKJ762IUN4A7UEZXLDERCVDW24QEKGEHFETAYZOKWQPVN2QJYA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.952636616 (469h11m41.823719661s) [noise=true , address=145.79.227.19:4079 , piece_id=ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.025177123 (469h11m41.896260168s) [noise=true , address=145.79.227.45:4229 , piece_id=GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.222526274 (469h11m42.093609319s) [noise=true , address=145.79.227.8:4331 , piece_id=ELLVNJFOVY5HSAHZDQYGQVO7OYHFZNFHCBJKHQFV5K6EV2M3JTRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.250024374 (469h11m42.121107409s) [noise=true , address=145.79.227.2:4114 , piece_id=5ISXZKQSH4CLLTF3ILGAXHV6STMLRUSZP6UATU2677X2FUPLBCRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.651787921 (469h11m42.522870966s) [noise=true , address=145.79.227.15:4875 , piece_id=FUMLT2ODHUE3ETSNWVKBYLSJXYVCHUW5KAJJE5AECNL3IAQYYCLQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.651814982 (469h11m42.522898027s) [seg_pos={0 0}, upload=411834]: Segment reached optimal threshold of 13 pieces. Dec 25 04:40:42.652052093 (469h11m42.523135129s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: write tcp 188.166.225.170:36650->77.90.31.15:4697: use of closed network connection; storage node overloaded, request limit: 2000 Dec 25 04:40:42.652069537 (469h11m42.523152572s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652177673 (469h11m42.523260708s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd): protocol: write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection; write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection Dec 25 04:40:42.652184456 (469h11m42.523267501s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652230243 (469h11m42.523313308s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq): protocol: write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection Dec 25 04:40:42.652240693 (469h11m42.523323778s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652268265 (469h11m42.52335131s) [seg_pos={0 0}, upload=411834]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 04:40:42.652354770 (469h11m42.523437815s) [upload=411834]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 04:40:42.652361693 (469h11m42.523444738s) [upload=411834]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 04:40:42.959704440 (469h11m42.830787485s) [upload=411834]: Done uploading segments... Dec 25 04:40:42.959728234 (469h11m42.83081127s) [upload=411834]: Upload finished. err: <nil> Duration: 5.876639201s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 5.702290396s trace: https://jaeger.dev.storj.io/trace/681cf842d2260d41 FAIL (2025-12-25 04:58:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, Piece ID: ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ) context deadline exceeded; piecestore: (Node ID: 1DT68m6J4PpDk3XUDhDaaqV46NLKVz4A23716JLFxEAh122GeR, Piece ID: WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA) context deadline exceeded; piecestore: (Node ID: 12wWYnhWHSqqAFgje4p9Cr7WYaJ6dJWsCqvrx8ZqVFoakjuFeka, Piece ID: GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q) context deadline exceeded ======LOGS====== nline: false Dec 25 04:40:39.039085560 (469h11m38.910168605s) [upload=411834]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 04:40:39.039095028 (469h11m38.910178073s) [upload=411834]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 04:40:39.297407613 (469h11m39.168490658s) [seg_pos={0 0}, upload=411834]: Begin upload segment... Dec 25 04:40:39.297471084 (469h11m39.168554119s) [seg_pos={0 0}, upload=411834]: Done begin upload segment. Dec 25 04:40:39.297511581 (469h11m39.168594616s) [noise=true , address=145.79.227.51:4411 , piece_id=WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.297663310 (469h11m39.168746345s) [noise=true , address=145.79.227.17:4970 , piece_id=6DPKJ762IUN4A7UEZXLDERCVDW24QEKGEHFETAYZOKWQPVN2QJYA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.298405574 (469h11m39.169488619s) [noise=true , address=145.79.227.2:4114 , piece_id=5ISXZKQSH4CLLTF3ILGAXHV6STMLRUSZP6UATU2677X2FUPLBCRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.298797961 (469h11m39.169881006s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299124392 (469h11m39.170207437s) [noise=true , address=45.91.116.5:4300 , piece_id=NTWC3RWYQQBBKPMXGUG44R2B5GWC2Z4LDFU3GTWOWCT3JIRIZOOQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299438650 (469h11m39.170521685s) [noise=true , address=145.79.227.5:4221 , piece_id=C4SXUVTG2TZ4EIRBGL4TLOLBMYOAZ4KLIQSZOXKEOWAJBQQJLWSA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299794709 (469h11m39.170877744s) [noise=true , address=145.79.227.37:4567 , piece_id=TZ4C57ISUI3CB3VOLTSM5W5QVDI7PVXV67PFJI3ZSVSXHTKPYPSA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.300142221 (469h11m39.171225256s) [noise=true , address=145.79.227.8:4331 , piece_id=ELLVNJFOVY5HSAHZDQYGQVO7OYHFZNFHCBJKHQFV5K6EV2M3JTRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.300506354 (469h11m39.171589389s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.301142795 (469h11m39.17222584s) [noise=true , address=145.79.227.36:4517 , piece_id=HBQU7CMAO2KJS6ODW2WHICBAWQSH6TQQPOQSENKDEISWCWFN2CMA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.301625835 (469h11m39.17270891s) [noise=true , address=145.79.227.45:4229 , piece_id=GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.302170964 (469h11m39.173254009s) [noise=true , address=145.79.227.3:4150 , piece_id=OHGA4EA33AVEDCJLB6QSUJLD3W2TV6YNN6FBVFFKTTSF2B6XDPRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.302629947 (469h11m39.173712982s) [noise=true , address=145.79.227.19:4079 , piece_id=ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303079704 (469h11m39.174162749s) [noise=true , address=145.79.227.15:4875 , piece_id=FUMLT2ODHUE3ETSNWVKBYLSJXYVCHUW5KAJJE5AECNL3IAQYYCLQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303499804 (469h11m39.174582849s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303921386 (469h11m39.175004432s) [noise=true , address=145.79.227.47:4265 , piece_id=TVUXNR6K35A3CWFDN4JRKM4Q6RGBC4LP2XKPXJFN2AJL32ESEBPA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:41.172453460 (469h11m41.043536495s) [upload=411834]: Next returned nil segment Dec 25 04:40:41.172473798 (469h11m41.043556833s) [upload=411834]: Waiting for error group managing segments... Dec 25 04:40:41.483361556 (469h11m41.354444601s) [noise=true , address=145.79.227.37:4567 , piece_id=TZ4C57ISUI3CB3VOLTSM5W5QVDI7PVXV67PFJI3ZSVSXHTKPYPSA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.484283170 (469h11m41.355366205s) [noise=true , address=145.79.227.5:4221 , piece_id=C4SXUVTG2TZ4EIRBGL4TLOLBMYOAZ4KLIQSZOXKEOWAJBQQJLWSA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.515393699 (469h11m41.386476744s) [noise=true , address=145.79.227.51:4411 , piece_id=WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.526179387 (469h11m41.397262422s) [noise=true , address=45.91.116.5:4300 , piece_id=NTWC3RWYQQBBKPMXGUG44R2B5GWC2Z4LDFU3GTWOWCT3JIRIZOOQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.642380892 (469h11m41.513463937s) [noise=true , address=145.79.227.36:4517 , piece_id=HBQU7CMAO2KJS6ODW2WHICBAWQSH6TQQPOQSENKDEISWCWFN2CMA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.662326662 (469h11m41.533409707s) [noise=true , address=145.79.227.3:4150 , piece_id=OHGA4EA33AVEDCJLB6QSUJLD3W2TV6YNN6FBVFFKTTSF2B6XDPRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.758852130 (469h11m41.629935175s) [noise=true , address=145.79.227.47:4265 , piece_id=TVUXNR6K35A3CWFDN4JRKM4Q6RGBC4LP2XKPXJFN2AJL32ESEBPA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.823449082 (469h11m41.694532117s) [noise=true , address=145.79.227.17:4970 , piece_id=6DPKJ762IUN4A7UEZXLDERCVDW24QEKGEHFETAYZOKWQPVN2QJYA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.952636616 (469h11m41.823719661s) [noise=true , address=145.79.227.19:4079 , piece_id=ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.025177123 (469h11m41.896260168s) [noise=true , address=145.79.227.45:4229 , piece_id=GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.222526274 (469h11m42.093609319s) [noise=true , address=145.79.227.8:4331 , piece_id=ELLVNJFOVY5HSAHZDQYGQVO7OYHFZNFHCBJKHQFV5K6EV2M3JTRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.250024374 (469h11m42.121107409s) [noise=true , address=145.79.227.2:4114 , piece_id=5ISXZKQSH4CLLTF3ILGAXHV6STMLRUSZP6UATU2677X2FUPLBCRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.651787921 (469h11m42.522870966s) [noise=true , address=145.79.227.15:4875 , piece_id=FUMLT2ODHUE3ETSNWVKBYLSJXYVCHUW5KAJJE5AECNL3IAQYYCLQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.651814982 (469h11m42.522898027s) [seg_pos={0 0}, upload=411834]: Segment reached optimal threshold of 13 pieces. Dec 25 04:40:42.652052093 (469h11m42.523135129s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: write tcp 188.166.225.170:36650->77.90.31.15:4697: use of closed network connection; storage node overloaded, request limit: 2000 Dec 25 04:40:42.652069537 (469h11m42.523152572s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652177673 (469h11m42.523260708s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd): protocol: write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection; write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection Dec 25 04:40:42.652184456 (469h11m42.523267501s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652230243 (469h11m42.523313308s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq): protocol: write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection Dec 25 04:40:42.652240693 (469h11m42.523323778s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652268265 (469h11m42.52335131s) [seg_pos={0 0}, upload=411834]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 04:40:42.652354770 (469h11m42.523437815s) [upload=411834]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 04:40:42.652361693 (469h11m42.523444738s) [upload=411834]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 04:40:42.959704440 (469h11m42.830787485s) [upload=411834]: Done uploading segments... Dec 25 04:40:42.959728234 (469h11m42.83081127s) [upload=411834]: Upload finished. err: <nil> Duration: 5.702319102s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 5.71683359s trace: https://jaeger.dev.storj.io/trace/681cf842d2260d41 FAIL (2025-12-25 04:57:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, Piece ID: ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ) context deadline exceeded; piecestore: (Node ID: 1DT68m6J4PpDk3XUDhDaaqV46NLKVz4A23716JLFxEAh122GeR, Piece ID: WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA) context deadline exceeded; piecestore: (Node ID: 12wWYnhWHSqqAFgje4p9Cr7WYaJ6dJWsCqvrx8ZqVFoakjuFeka, Piece ID: GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q) context deadline exceeded ======LOGS====== nline: false Dec 25 04:40:39.039085560 (469h11m38.910168605s) [upload=411834]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 04:40:39.039095028 (469h11m38.910178073s) [upload=411834]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 04:40:39.297407613 (469h11m39.168490658s) [seg_pos={0 0}, upload=411834]: Begin upload segment... Dec 25 04:40:39.297471084 (469h11m39.168554119s) [seg_pos={0 0}, upload=411834]: Done begin upload segment. Dec 25 04:40:39.297511581 (469h11m39.168594616s) [noise=true , address=145.79.227.51:4411 , piece_id=WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.297663310 (469h11m39.168746345s) [noise=true , address=145.79.227.17:4970 , piece_id=6DPKJ762IUN4A7UEZXLDERCVDW24QEKGEHFETAYZOKWQPVN2QJYA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.298405574 (469h11m39.169488619s) [noise=true , address=145.79.227.2:4114 , piece_id=5ISXZKQSH4CLLTF3ILGAXHV6STMLRUSZP6UATU2677X2FUPLBCRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.298797961 (469h11m39.169881006s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299124392 (469h11m39.170207437s) [noise=true , address=45.91.116.5:4300 , piece_id=NTWC3RWYQQBBKPMXGUG44R2B5GWC2Z4LDFU3GTWOWCT3JIRIZOOQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299438650 (469h11m39.170521685s) [noise=true , address=145.79.227.5:4221 , piece_id=C4SXUVTG2TZ4EIRBGL4TLOLBMYOAZ4KLIQSZOXKEOWAJBQQJLWSA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299794709 (469h11m39.170877744s) [noise=true , address=145.79.227.37:4567 , piece_id=TZ4C57ISUI3CB3VOLTSM5W5QVDI7PVXV67PFJI3ZSVSXHTKPYPSA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.300142221 (469h11m39.171225256s) [noise=true , address=145.79.227.8:4331 , piece_id=ELLVNJFOVY5HSAHZDQYGQVO7OYHFZNFHCBJKHQFV5K6EV2M3JTRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.300506354 (469h11m39.171589389s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.301142795 (469h11m39.17222584s) [noise=true , address=145.79.227.36:4517 , piece_id=HBQU7CMAO2KJS6ODW2WHICBAWQSH6TQQPOQSENKDEISWCWFN2CMA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.301625835 (469h11m39.17270891s) [noise=true , address=145.79.227.45:4229 , piece_id=GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.302170964 (469h11m39.173254009s) [noise=true , address=145.79.227.3:4150 , piece_id=OHGA4EA33AVEDCJLB6QSUJLD3W2TV6YNN6FBVFFKTTSF2B6XDPRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.302629947 (469h11m39.173712982s) [noise=true , address=145.79.227.19:4079 , piece_id=ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303079704 (469h11m39.174162749s) [noise=true , address=145.79.227.15:4875 , piece_id=FUMLT2ODHUE3ETSNWVKBYLSJXYVCHUW5KAJJE5AECNL3IAQYYCLQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303499804 (469h11m39.174582849s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303921386 (469h11m39.175004432s) [noise=true , address=145.79.227.47:4265 , piece_id=TVUXNR6K35A3CWFDN4JRKM4Q6RGBC4LP2XKPXJFN2AJL32ESEBPA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:41.172453460 (469h11m41.043536495s) [upload=411834]: Next returned nil segment Dec 25 04:40:41.172473798 (469h11m41.043556833s) [upload=411834]: Waiting for error group managing segments... Dec 25 04:40:41.483361556 (469h11m41.354444601s) [noise=true , address=145.79.227.37:4567 , piece_id=TZ4C57ISUI3CB3VOLTSM5W5QVDI7PVXV67PFJI3ZSVSXHTKPYPSA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.484283170 (469h11m41.355366205s) [noise=true , address=145.79.227.5:4221 , piece_id=C4SXUVTG2TZ4EIRBGL4TLOLBMYOAZ4KLIQSZOXKEOWAJBQQJLWSA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.515393699 (469h11m41.386476744s) [noise=true , address=145.79.227.51:4411 , piece_id=WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.526179387 (469h11m41.397262422s) [noise=true , address=45.91.116.5:4300 , piece_id=NTWC3RWYQQBBKPMXGUG44R2B5GWC2Z4LDFU3GTWOWCT3JIRIZOOQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.642380892 (469h11m41.513463937s) [noise=true , address=145.79.227.36:4517 , piece_id=HBQU7CMAO2KJS6ODW2WHICBAWQSH6TQQPOQSENKDEISWCWFN2CMA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.662326662 (469h11m41.533409707s) [noise=true , address=145.79.227.3:4150 , piece_id=OHGA4EA33AVEDCJLB6QSUJLD3W2TV6YNN6FBVFFKTTSF2B6XDPRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.758852130 (469h11m41.629935175s) [noise=true , address=145.79.227.47:4265 , piece_id=TVUXNR6K35A3CWFDN4JRKM4Q6RGBC4LP2XKPXJFN2AJL32ESEBPA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.823449082 (469h11m41.694532117s) [noise=true , address=145.79.227.17:4970 , piece_id=6DPKJ762IUN4A7UEZXLDERCVDW24QEKGEHFETAYZOKWQPVN2QJYA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.952636616 (469h11m41.823719661s) [noise=true , address=145.79.227.19:4079 , piece_id=ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.025177123 (469h11m41.896260168s) [noise=true , address=145.79.227.45:4229 , piece_id=GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.222526274 (469h11m42.093609319s) [noise=true , address=145.79.227.8:4331 , piece_id=ELLVNJFOVY5HSAHZDQYGQVO7OYHFZNFHCBJKHQFV5K6EV2M3JTRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.250024374 (469h11m42.121107409s) [noise=true , address=145.79.227.2:4114 , piece_id=5ISXZKQSH4CLLTF3ILGAXHV6STMLRUSZP6UATU2677X2FUPLBCRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.651787921 (469h11m42.522870966s) [noise=true , address=145.79.227.15:4875 , piece_id=FUMLT2ODHUE3ETSNWVKBYLSJXYVCHUW5KAJJE5AECNL3IAQYYCLQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.651814982 (469h11m42.522898027s) [seg_pos={0 0}, upload=411834]: Segment reached optimal threshold of 13 pieces. Dec 25 04:40:42.652052093 (469h11m42.523135129s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: write tcp 188.166.225.170:36650->77.90.31.15:4697: use of closed network connection; storage node overloaded, request limit: 2000 Dec 25 04:40:42.652069537 (469h11m42.523152572s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652177673 (469h11m42.523260708s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd): protocol: write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection; write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection Dec 25 04:40:42.652184456 (469h11m42.523267501s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652230243 (469h11m42.523313308s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq): protocol: write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection Dec 25 04:40:42.652240693 (469h11m42.523323778s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652268265 (469h11m42.52335131s) [seg_pos={0 0}, upload=411834]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 04:40:42.652354770 (469h11m42.523437815s) [upload=411834]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 04:40:42.652361693 (469h11m42.523444738s) [upload=411834]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 04:40:42.959704440 (469h11m42.830787485s) [upload=411834]: Done uploading segments... Dec 25 04:40:42.959728234 (469h11m42.83081127s) [upload=411834]: Upload finished. err: <nil> Duration: 5.716852856s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 6.055676945s trace: https://jaeger.dev.storj.io/trace/681cf842d2260d41 FAIL (2025-12-25 04:57:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, Piece ID: ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ) context deadline exceeded; piecestore: (Node ID: 1DT68m6J4PpDk3XUDhDaaqV46NLKVz4A23716JLFxEAh122GeR, Piece ID: WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA) context deadline exceeded; piecestore: (Node ID: 12wWYnhWHSqqAFgje4p9Cr7WYaJ6dJWsCqvrx8ZqVFoakjuFeka, Piece ID: GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q) context deadline exceeded ======LOGS====== nline: false Dec 25 04:40:39.039085560 (469h11m38.910168605s) [upload=411834]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 04:40:39.039095028 (469h11m38.910178073s) [upload=411834]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 04:40:39.297407613 (469h11m39.168490658s) [seg_pos={0 0}, upload=411834]: Begin upload segment... Dec 25 04:40:39.297471084 (469h11m39.168554119s) [seg_pos={0 0}, upload=411834]: Done begin upload segment. Dec 25 04:40:39.297511581 (469h11m39.168594616s) [noise=true , address=145.79.227.51:4411 , piece_id=WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.297663310 (469h11m39.168746345s) [noise=true , address=145.79.227.17:4970 , piece_id=6DPKJ762IUN4A7UEZXLDERCVDW24QEKGEHFETAYZOKWQPVN2QJYA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.298405574 (469h11m39.169488619s) [noise=true , address=145.79.227.2:4114 , piece_id=5ISXZKQSH4CLLTF3ILGAXHV6STMLRUSZP6UATU2677X2FUPLBCRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.298797961 (469h11m39.169881006s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299124392 (469h11m39.170207437s) [noise=true , address=45.91.116.5:4300 , piece_id=NTWC3RWYQQBBKPMXGUG44R2B5GWC2Z4LDFU3GTWOWCT3JIRIZOOQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299438650 (469h11m39.170521685s) [noise=true , address=145.79.227.5:4221 , piece_id=C4SXUVTG2TZ4EIRBGL4TLOLBMYOAZ4KLIQSZOXKEOWAJBQQJLWSA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299794709 (469h11m39.170877744s) [noise=true , address=145.79.227.37:4567 , piece_id=TZ4C57ISUI3CB3VOLTSM5W5QVDI7PVXV67PFJI3ZSVSXHTKPYPSA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.300142221 (469h11m39.171225256s) [noise=true , address=145.79.227.8:4331 , piece_id=ELLVNJFOVY5HSAHZDQYGQVO7OYHFZNFHCBJKHQFV5K6EV2M3JTRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.300506354 (469h11m39.171589389s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.301142795 (469h11m39.17222584s) [noise=true , address=145.79.227.36:4517 , piece_id=HBQU7CMAO2KJS6ODW2WHICBAWQSH6TQQPOQSENKDEISWCWFN2CMA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.301625835 (469h11m39.17270891s) [noise=true , address=145.79.227.45:4229 , piece_id=GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.302170964 (469h11m39.173254009s) [noise=true , address=145.79.227.3:4150 , piece_id=OHGA4EA33AVEDCJLB6QSUJLD3W2TV6YNN6FBVFFKTTSF2B6XDPRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.302629947 (469h11m39.173712982s) [noise=true , address=145.79.227.19:4079 , piece_id=ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303079704 (469h11m39.174162749s) [noise=true , address=145.79.227.15:4875 , piece_id=FUMLT2ODHUE3ETSNWVKBYLSJXYVCHUW5KAJJE5AECNL3IAQYYCLQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303499804 (469h11m39.174582849s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303921386 (469h11m39.175004432s) [noise=true , address=145.79.227.47:4265 , piece_id=TVUXNR6K35A3CWFDN4JRKM4Q6RGBC4LP2XKPXJFN2AJL32ESEBPA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:41.172453460 (469h11m41.043536495s) [upload=411834]: Next returned nil segment Dec 25 04:40:41.172473798 (469h11m41.043556833s) [upload=411834]: Waiting for error group managing segments... Dec 25 04:40:41.483361556 (469h11m41.354444601s) [noise=true , address=145.79.227.37:4567 , piece_id=TZ4C57ISUI3CB3VOLTSM5W5QVDI7PVXV67PFJI3ZSVSXHTKPYPSA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.484283170 (469h11m41.355366205s) [noise=true , address=145.79.227.5:4221 , piece_id=C4SXUVTG2TZ4EIRBGL4TLOLBMYOAZ4KLIQSZOXKEOWAJBQQJLWSA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.515393699 (469h11m41.386476744s) [noise=true , address=145.79.227.51:4411 , piece_id=WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.526179387 (469h11m41.397262422s) [noise=true , address=45.91.116.5:4300 , piece_id=NTWC3RWYQQBBKPMXGUG44R2B5GWC2Z4LDFU3GTWOWCT3JIRIZOOQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.642380892 (469h11m41.513463937s) [noise=true , address=145.79.227.36:4517 , piece_id=HBQU7CMAO2KJS6ODW2WHICBAWQSH6TQQPOQSENKDEISWCWFN2CMA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.662326662 (469h11m41.533409707s) [noise=true , address=145.79.227.3:4150 , piece_id=OHGA4EA33AVEDCJLB6QSUJLD3W2TV6YNN6FBVFFKTTSF2B6XDPRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.758852130 (469h11m41.629935175s) [noise=true , address=145.79.227.47:4265 , piece_id=TVUXNR6K35A3CWFDN4JRKM4Q6RGBC4LP2XKPXJFN2AJL32ESEBPA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.823449082 (469h11m41.694532117s) [noise=true , address=145.79.227.17:4970 , piece_id=6DPKJ762IUN4A7UEZXLDERCVDW24QEKGEHFETAYZOKWQPVN2QJYA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.952636616 (469h11m41.823719661s) [noise=true , address=145.79.227.19:4079 , piece_id=ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.025177123 (469h11m41.896260168s) [noise=true , address=145.79.227.45:4229 , piece_id=GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.222526274 (469h11m42.093609319s) [noise=true , address=145.79.227.8:4331 , piece_id=ELLVNJFOVY5HSAHZDQYGQVO7OYHFZNFHCBJKHQFV5K6EV2M3JTRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.250024374 (469h11m42.121107409s) [noise=true , address=145.79.227.2:4114 , piece_id=5ISXZKQSH4CLLTF3ILGAXHV6STMLRUSZP6UATU2677X2FUPLBCRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.651787921 (469h11m42.522870966s) [noise=true , address=145.79.227.15:4875 , piece_id=FUMLT2ODHUE3ETSNWVKBYLSJXYVCHUW5KAJJE5AECNL3IAQYYCLQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.651814982 (469h11m42.522898027s) [seg_pos={0 0}, upload=411834]: Segment reached optimal threshold of 13 pieces. Dec 25 04:40:42.652052093 (469h11m42.523135129s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: write tcp 188.166.225.170:36650->77.90.31.15:4697: use of closed network connection; storage node overloaded, request limit: 2000 Dec 25 04:40:42.652069537 (469h11m42.523152572s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652177673 (469h11m42.523260708s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd): protocol: write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection; write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection Dec 25 04:40:42.652184456 (469h11m42.523267501s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652230243 (469h11m42.523313308s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq): protocol: write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection Dec 25 04:40:42.652240693 (469h11m42.523323778s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652268265 (469h11m42.52335131s) [seg_pos={0 0}, upload=411834]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 04:40:42.652354770 (469h11m42.523437815s) [upload=411834]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 04:40:42.652361693 (469h11m42.523444738s) [upload=411834]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 04:40:42.959704440 (469h11m42.830787485s) [upload=411834]: Done uploading segments... Dec 25 04:40:42.959728234 (469h11m42.83081127s) [upload=411834]: Upload finished. err: <nil> Duration: 6.055708274s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 5.815722255s trace: https://jaeger.dev.storj.io/trace/681cf842d2260d41 FAIL (2025-12-25 04:56:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, Piece ID: ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ) context deadline exceeded; piecestore: (Node ID: 1DT68m6J4PpDk3XUDhDaaqV46NLKVz4A23716JLFxEAh122GeR, Piece ID: WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA) context deadline exceeded; piecestore: (Node ID: 12wWYnhWHSqqAFgje4p9Cr7WYaJ6dJWsCqvrx8ZqVFoakjuFeka, Piece ID: GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q) context deadline exceeded ======LOGS====== nline: false Dec 25 04:40:39.039085560 (469h11m38.910168605s) [upload=411834]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 04:40:39.039095028 (469h11m38.910178073s) [upload=411834]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 04:40:39.297407613 (469h11m39.168490658s) [seg_pos={0 0}, upload=411834]: Begin upload segment... Dec 25 04:40:39.297471084 (469h11m39.168554119s) [seg_pos={0 0}, upload=411834]: Done begin upload segment. Dec 25 04:40:39.297511581 (469h11m39.168594616s) [noise=true , address=145.79.227.51:4411 , piece_id=WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.297663310 (469h11m39.168746345s) [noise=true , address=145.79.227.17:4970 , piece_id=6DPKJ762IUN4A7UEZXLDERCVDW24QEKGEHFETAYZOKWQPVN2QJYA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.298405574 (469h11m39.169488619s) [noise=true , address=145.79.227.2:4114 , piece_id=5ISXZKQSH4CLLTF3ILGAXHV6STMLRUSZP6UATU2677X2FUPLBCRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.298797961 (469h11m39.169881006s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299124392 (469h11m39.170207437s) [noise=true , address=45.91.116.5:4300 , piece_id=NTWC3RWYQQBBKPMXGUG44R2B5GWC2Z4LDFU3GTWOWCT3JIRIZOOQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299438650 (469h11m39.170521685s) [noise=true , address=145.79.227.5:4221 , piece_id=C4SXUVTG2TZ4EIRBGL4TLOLBMYOAZ4KLIQSZOXKEOWAJBQQJLWSA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299794709 (469h11m39.170877744s) [noise=true , address=145.79.227.37:4567 , piece_id=TZ4C57ISUI3CB3VOLTSM5W5QVDI7PVXV67PFJI3ZSVSXHTKPYPSA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.300142221 (469h11m39.171225256s) [noise=true , address=145.79.227.8:4331 , piece_id=ELLVNJFOVY5HSAHZDQYGQVO7OYHFZNFHCBJKHQFV5K6EV2M3JTRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.300506354 (469h11m39.171589389s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.301142795 (469h11m39.17222584s) [noise=true , address=145.79.227.36:4517 , piece_id=HBQU7CMAO2KJS6ODW2WHICBAWQSH6TQQPOQSENKDEISWCWFN2CMA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.301625835 (469h11m39.17270891s) [noise=true , address=145.79.227.45:4229 , piece_id=GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.302170964 (469h11m39.173254009s) [noise=true , address=145.79.227.3:4150 , piece_id=OHGA4EA33AVEDCJLB6QSUJLD3W2TV6YNN6FBVFFKTTSF2B6XDPRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.302629947 (469h11m39.173712982s) [noise=true , address=145.79.227.19:4079 , piece_id=ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303079704 (469h11m39.174162749s) [noise=true , address=145.79.227.15:4875 , piece_id=FUMLT2ODHUE3ETSNWVKBYLSJXYVCHUW5KAJJE5AECNL3IAQYYCLQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303499804 (469h11m39.174582849s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303921386 (469h11m39.175004432s) [noise=true , address=145.79.227.47:4265 , piece_id=TVUXNR6K35A3CWFDN4JRKM4Q6RGBC4LP2XKPXJFN2AJL32ESEBPA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:41.172453460 (469h11m41.043536495s) [upload=411834]: Next returned nil segment Dec 25 04:40:41.172473798 (469h11m41.043556833s) [upload=411834]: Waiting for error group managing segments... Dec 25 04:40:41.483361556 (469h11m41.354444601s) [noise=true , address=145.79.227.37:4567 , piece_id=TZ4C57ISUI3CB3VOLTSM5W5QVDI7PVXV67PFJI3ZSVSXHTKPYPSA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.484283170 (469h11m41.355366205s) [noise=true , address=145.79.227.5:4221 , piece_id=C4SXUVTG2TZ4EIRBGL4TLOLBMYOAZ4KLIQSZOXKEOWAJBQQJLWSA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.515393699 (469h11m41.386476744s) [noise=true , address=145.79.227.51:4411 , piece_id=WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.526179387 (469h11m41.397262422s) [noise=true , address=45.91.116.5:4300 , piece_id=NTWC3RWYQQBBKPMXGUG44R2B5GWC2Z4LDFU3GTWOWCT3JIRIZOOQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.642380892 (469h11m41.513463937s) [noise=true , address=145.79.227.36:4517 , piece_id=HBQU7CMAO2KJS6ODW2WHICBAWQSH6TQQPOQSENKDEISWCWFN2CMA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.662326662 (469h11m41.533409707s) [noise=true , address=145.79.227.3:4150 , piece_id=OHGA4EA33AVEDCJLB6QSUJLD3W2TV6YNN6FBVFFKTTSF2B6XDPRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.758852130 (469h11m41.629935175s) [noise=true , address=145.79.227.47:4265 , piece_id=TVUXNR6K35A3CWFDN4JRKM4Q6RGBC4LP2XKPXJFN2AJL32ESEBPA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.823449082 (469h11m41.694532117s) [noise=true , address=145.79.227.17:4970 , piece_id=6DPKJ762IUN4A7UEZXLDERCVDW24QEKGEHFETAYZOKWQPVN2QJYA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.952636616 (469h11m41.823719661s) [noise=true , address=145.79.227.19:4079 , piece_id=ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.025177123 (469h11m41.896260168s) [noise=true , address=145.79.227.45:4229 , piece_id=GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.222526274 (469h11m42.093609319s) [noise=true , address=145.79.227.8:4331 , piece_id=ELLVNJFOVY5HSAHZDQYGQVO7OYHFZNFHCBJKHQFV5K6EV2M3JTRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.250024374 (469h11m42.121107409s) [noise=true , address=145.79.227.2:4114 , piece_id=5ISXZKQSH4CLLTF3ILGAXHV6STMLRUSZP6UATU2677X2FUPLBCRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.651787921 (469h11m42.522870966s) [noise=true , address=145.79.227.15:4875 , piece_id=FUMLT2ODHUE3ETSNWVKBYLSJXYVCHUW5KAJJE5AECNL3IAQYYCLQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.651814982 (469h11m42.522898027s) [seg_pos={0 0}, upload=411834]: Segment reached optimal threshold of 13 pieces. Dec 25 04:40:42.652052093 (469h11m42.523135129s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: write tcp 188.166.225.170:36650->77.90.31.15:4697: use of closed network connection; storage node overloaded, request limit: 2000 Dec 25 04:40:42.652069537 (469h11m42.523152572s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652177673 (469h11m42.523260708s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd): protocol: write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection; write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection Dec 25 04:40:42.652184456 (469h11m42.523267501s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652230243 (469h11m42.523313308s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq): protocol: write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection Dec 25 04:40:42.652240693 (469h11m42.523323778s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652268265 (469h11m42.52335131s) [seg_pos={0 0}, upload=411834]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 04:40:42.652354770 (469h11m42.523437815s) [upload=411834]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 04:40:42.652361693 (469h11m42.523444738s) [upload=411834]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 04:40:42.959704440 (469h11m42.830787485s) [upload=411834]: Done uploading segments... Dec 25 04:40:42.959728234 (469h11m42.83081127s) [upload=411834]: Upload finished. err: <nil> Duration: 5.815796647s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 5.851006504s trace: https://jaeger.dev.storj.io/trace/681cf842d2260d41 FAIL (2025-12-25 04:56:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, Piece ID: ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ) context deadline exceeded; piecestore: (Node ID: 1DT68m6J4PpDk3XUDhDaaqV46NLKVz4A23716JLFxEAh122GeR, Piece ID: WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA) context deadline exceeded; piecestore: (Node ID: 12wWYnhWHSqqAFgje4p9Cr7WYaJ6dJWsCqvrx8ZqVFoakjuFeka, Piece ID: GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q) context deadline exceeded ======LOGS====== nline: false Dec 25 04:40:39.039085560 (469h11m38.910168605s) [upload=411834]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 04:40:39.039095028 (469h11m38.910178073s) [upload=411834]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 04:40:39.297407613 (469h11m39.168490658s) [seg_pos={0 0}, upload=411834]: Begin upload segment... Dec 25 04:40:39.297471084 (469h11m39.168554119s) [seg_pos={0 0}, upload=411834]: Done begin upload segment. Dec 25 04:40:39.297511581 (469h11m39.168594616s) [noise=true , address=145.79.227.51:4411 , piece_id=WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.297663310 (469h11m39.168746345s) [noise=true , address=145.79.227.17:4970 , piece_id=6DPKJ762IUN4A7UEZXLDERCVDW24QEKGEHFETAYZOKWQPVN2QJYA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.298405574 (469h11m39.169488619s) [noise=true , address=145.79.227.2:4114 , piece_id=5ISXZKQSH4CLLTF3ILGAXHV6STMLRUSZP6UATU2677X2FUPLBCRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.298797961 (469h11m39.169881006s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299124392 (469h11m39.170207437s) [noise=true , address=45.91.116.5:4300 , piece_id=NTWC3RWYQQBBKPMXGUG44R2B5GWC2Z4LDFU3GTWOWCT3JIRIZOOQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299438650 (469h11m39.170521685s) [noise=true , address=145.79.227.5:4221 , piece_id=C4SXUVTG2TZ4EIRBGL4TLOLBMYOAZ4KLIQSZOXKEOWAJBQQJLWSA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299794709 (469h11m39.170877744s) [noise=true , address=145.79.227.37:4567 , piece_id=TZ4C57ISUI3CB3VOLTSM5W5QVDI7PVXV67PFJI3ZSVSXHTKPYPSA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.300142221 (469h11m39.171225256s) [noise=true , address=145.79.227.8:4331 , piece_id=ELLVNJFOVY5HSAHZDQYGQVO7OYHFZNFHCBJKHQFV5K6EV2M3JTRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.300506354 (469h11m39.171589389s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.301142795 (469h11m39.17222584s) [noise=true , address=145.79.227.36:4517 , piece_id=HBQU7CMAO2KJS6ODW2WHICBAWQSH6TQQPOQSENKDEISWCWFN2CMA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.301625835 (469h11m39.17270891s) [noise=true , address=145.79.227.45:4229 , piece_id=GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.302170964 (469h11m39.173254009s) [noise=true , address=145.79.227.3:4150 , piece_id=OHGA4EA33AVEDCJLB6QSUJLD3W2TV6YNN6FBVFFKTTSF2B6XDPRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.302629947 (469h11m39.173712982s) [noise=true , address=145.79.227.19:4079 , piece_id=ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303079704 (469h11m39.174162749s) [noise=true , address=145.79.227.15:4875 , piece_id=FUMLT2ODHUE3ETSNWVKBYLSJXYVCHUW5KAJJE5AECNL3IAQYYCLQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303499804 (469h11m39.174582849s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303921386 (469h11m39.175004432s) [noise=true , address=145.79.227.47:4265 , piece_id=TVUXNR6K35A3CWFDN4JRKM4Q6RGBC4LP2XKPXJFN2AJL32ESEBPA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:41.172453460 (469h11m41.043536495s) [upload=411834]: Next returned nil segment Dec 25 04:40:41.172473798 (469h11m41.043556833s) [upload=411834]: Waiting for error group managing segments... Dec 25 04:40:41.483361556 (469h11m41.354444601s) [noise=true , address=145.79.227.37:4567 , piece_id=TZ4C57ISUI3CB3VOLTSM5W5QVDI7PVXV67PFJI3ZSVSXHTKPYPSA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.484283170 (469h11m41.355366205s) [noise=true , address=145.79.227.5:4221 , piece_id=C4SXUVTG2TZ4EIRBGL4TLOLBMYOAZ4KLIQSZOXKEOWAJBQQJLWSA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.515393699 (469h11m41.386476744s) [noise=true , address=145.79.227.51:4411 , piece_id=WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.526179387 (469h11m41.397262422s) [noise=true , address=45.91.116.5:4300 , piece_id=NTWC3RWYQQBBKPMXGUG44R2B5GWC2Z4LDFU3GTWOWCT3JIRIZOOQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.642380892 (469h11m41.513463937s) [noise=true , address=145.79.227.36:4517 , piece_id=HBQU7CMAO2KJS6ODW2WHICBAWQSH6TQQPOQSENKDEISWCWFN2CMA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.662326662 (469h11m41.533409707s) [noise=true , address=145.79.227.3:4150 , piece_id=OHGA4EA33AVEDCJLB6QSUJLD3W2TV6YNN6FBVFFKTTSF2B6XDPRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.758852130 (469h11m41.629935175s) [noise=true , address=145.79.227.47:4265 , piece_id=TVUXNR6K35A3CWFDN4JRKM4Q6RGBC4LP2XKPXJFN2AJL32ESEBPA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.823449082 (469h11m41.694532117s) [noise=true , address=145.79.227.17:4970 , piece_id=6DPKJ762IUN4A7UEZXLDERCVDW24QEKGEHFETAYZOKWQPVN2QJYA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.952636616 (469h11m41.823719661s) [noise=true , address=145.79.227.19:4079 , piece_id=ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.025177123 (469h11m41.896260168s) [noise=true , address=145.79.227.45:4229 , piece_id=GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.222526274 (469h11m42.093609319s) [noise=true , address=145.79.227.8:4331 , piece_id=ELLVNJFOVY5HSAHZDQYGQVO7OYHFZNFHCBJKHQFV5K6EV2M3JTRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.250024374 (469h11m42.121107409s) [noise=true , address=145.79.227.2:4114 , piece_id=5ISXZKQSH4CLLTF3ILGAXHV6STMLRUSZP6UATU2677X2FUPLBCRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.651787921 (469h11m42.522870966s) [noise=true , address=145.79.227.15:4875 , piece_id=FUMLT2ODHUE3ETSNWVKBYLSJXYVCHUW5KAJJE5AECNL3IAQYYCLQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.651814982 (469h11m42.522898027s) [seg_pos={0 0}, upload=411834]: Segment reached optimal threshold of 13 pieces. Dec 25 04:40:42.652052093 (469h11m42.523135129s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: write tcp 188.166.225.170:36650->77.90.31.15:4697: use of closed network connection; storage node overloaded, request limit: 2000 Dec 25 04:40:42.652069537 (469h11m42.523152572s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652177673 (469h11m42.523260708s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd): protocol: write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection; write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection Dec 25 04:40:42.652184456 (469h11m42.523267501s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652230243 (469h11m42.523313308s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq): protocol: write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection Dec 25 04:40:42.652240693 (469h11m42.523323778s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652268265 (469h11m42.52335131s) [seg_pos={0 0}, upload=411834]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 04:40:42.652354770 (469h11m42.523437815s) [upload=411834]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 04:40:42.652361693 (469h11m42.523444738s) [upload=411834]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 04:40:42.959704440 (469h11m42.830787485s) [upload=411834]: Done uploading segments... Dec 25 04:40:42.959728234 (469h11m42.83081127s) [upload=411834]: Upload finished. err: <nil> Duration: 5.851036309s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 6.051644742s trace: https://jaeger.dev.storj.io/trace/681cf842d2260d41 FAIL (2025-12-25 04:55:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, Piece ID: ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ) context deadline exceeded; piecestore: (Node ID: 1DT68m6J4PpDk3XUDhDaaqV46NLKVz4A23716JLFxEAh122GeR, Piece ID: WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA) context deadline exceeded; piecestore: (Node ID: 12wWYnhWHSqqAFgje4p9Cr7WYaJ6dJWsCqvrx8ZqVFoakjuFeka, Piece ID: GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q) context deadline exceeded ======LOGS====== nline: false Dec 25 04:40:39.039085560 (469h11m38.910168605s) [upload=411834]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 04:40:39.039095028 (469h11m38.910178073s) [upload=411834]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 04:40:39.297407613 (469h11m39.168490658s) [seg_pos={0 0}, upload=411834]: Begin upload segment... Dec 25 04:40:39.297471084 (469h11m39.168554119s) [seg_pos={0 0}, upload=411834]: Done begin upload segment. Dec 25 04:40:39.297511581 (469h11m39.168594616s) [noise=true , address=145.79.227.51:4411 , piece_id=WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.297663310 (469h11m39.168746345s) [noise=true , address=145.79.227.17:4970 , piece_id=6DPKJ762IUN4A7UEZXLDERCVDW24QEKGEHFETAYZOKWQPVN2QJYA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.298405574 (469h11m39.169488619s) [noise=true , address=145.79.227.2:4114 , piece_id=5ISXZKQSH4CLLTF3ILGAXHV6STMLRUSZP6UATU2677X2FUPLBCRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.298797961 (469h11m39.169881006s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299124392 (469h11m39.170207437s) [noise=true , address=45.91.116.5:4300 , piece_id=NTWC3RWYQQBBKPMXGUG44R2B5GWC2Z4LDFU3GTWOWCT3JIRIZOOQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299438650 (469h11m39.170521685s) [noise=true , address=145.79.227.5:4221 , piece_id=C4SXUVTG2TZ4EIRBGL4TLOLBMYOAZ4KLIQSZOXKEOWAJBQQJLWSA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299794709 (469h11m39.170877744s) [noise=true , address=145.79.227.37:4567 , piece_id=TZ4C57ISUI3CB3VOLTSM5W5QVDI7PVXV67PFJI3ZSVSXHTKPYPSA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.300142221 (469h11m39.171225256s) [noise=true , address=145.79.227.8:4331 , piece_id=ELLVNJFOVY5HSAHZDQYGQVO7OYHFZNFHCBJKHQFV5K6EV2M3JTRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.300506354 (469h11m39.171589389s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.301142795 (469h11m39.17222584s) [noise=true , address=145.79.227.36:4517 , piece_id=HBQU7CMAO2KJS6ODW2WHICBAWQSH6TQQPOQSENKDEISWCWFN2CMA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.301625835 (469h11m39.17270891s) [noise=true , address=145.79.227.45:4229 , piece_id=GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.302170964 (469h11m39.173254009s) [noise=true , address=145.79.227.3:4150 , piece_id=OHGA4EA33AVEDCJLB6QSUJLD3W2TV6YNN6FBVFFKTTSF2B6XDPRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.302629947 (469h11m39.173712982s) [noise=true , address=145.79.227.19:4079 , piece_id=ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303079704 (469h11m39.174162749s) [noise=true , address=145.79.227.15:4875 , piece_id=FUMLT2ODHUE3ETSNWVKBYLSJXYVCHUW5KAJJE5AECNL3IAQYYCLQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303499804 (469h11m39.174582849s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303921386 (469h11m39.175004432s) [noise=true , address=145.79.227.47:4265 , piece_id=TVUXNR6K35A3CWFDN4JRKM4Q6RGBC4LP2XKPXJFN2AJL32ESEBPA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:41.172453460 (469h11m41.043536495s) [upload=411834]: Next returned nil segment Dec 25 04:40:41.172473798 (469h11m41.043556833s) [upload=411834]: Waiting for error group managing segments... Dec 25 04:40:41.483361556 (469h11m41.354444601s) [noise=true , address=145.79.227.37:4567 , piece_id=TZ4C57ISUI3CB3VOLTSM5W5QVDI7PVXV67PFJI3ZSVSXHTKPYPSA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.484283170 (469h11m41.355366205s) [noise=true , address=145.79.227.5:4221 , piece_id=C4SXUVTG2TZ4EIRBGL4TLOLBMYOAZ4KLIQSZOXKEOWAJBQQJLWSA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.515393699 (469h11m41.386476744s) [noise=true , address=145.79.227.51:4411 , piece_id=WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.526179387 (469h11m41.397262422s) [noise=true , address=45.91.116.5:4300 , piece_id=NTWC3RWYQQBBKPMXGUG44R2B5GWC2Z4LDFU3GTWOWCT3JIRIZOOQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.642380892 (469h11m41.513463937s) [noise=true , address=145.79.227.36:4517 , piece_id=HBQU7CMAO2KJS6ODW2WHICBAWQSH6TQQPOQSENKDEISWCWFN2CMA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.662326662 (469h11m41.533409707s) [noise=true , address=145.79.227.3:4150 , piece_id=OHGA4EA33AVEDCJLB6QSUJLD3W2TV6YNN6FBVFFKTTSF2B6XDPRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.758852130 (469h11m41.629935175s) [noise=true , address=145.79.227.47:4265 , piece_id=TVUXNR6K35A3CWFDN4JRKM4Q6RGBC4LP2XKPXJFN2AJL32ESEBPA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.823449082 (469h11m41.694532117s) [noise=true , address=145.79.227.17:4970 , piece_id=6DPKJ762IUN4A7UEZXLDERCVDW24QEKGEHFETAYZOKWQPVN2QJYA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.952636616 (469h11m41.823719661s) [noise=true , address=145.79.227.19:4079 , piece_id=ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.025177123 (469h11m41.896260168s) [noise=true , address=145.79.227.45:4229 , piece_id=GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.222526274 (469h11m42.093609319s) [noise=true , address=145.79.227.8:4331 , piece_id=ELLVNJFOVY5HSAHZDQYGQVO7OYHFZNFHCBJKHQFV5K6EV2M3JTRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.250024374 (469h11m42.121107409s) [noise=true , address=145.79.227.2:4114 , piece_id=5ISXZKQSH4CLLTF3ILGAXHV6STMLRUSZP6UATU2677X2FUPLBCRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.651787921 (469h11m42.522870966s) [noise=true , address=145.79.227.15:4875 , piece_id=FUMLT2ODHUE3ETSNWVKBYLSJXYVCHUW5KAJJE5AECNL3IAQYYCLQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.651814982 (469h11m42.522898027s) [seg_pos={0 0}, upload=411834]: Segment reached optimal threshold of 13 pieces. Dec 25 04:40:42.652052093 (469h11m42.523135129s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: write tcp 188.166.225.170:36650->77.90.31.15:4697: use of closed network connection; storage node overloaded, request limit: 2000 Dec 25 04:40:42.652069537 (469h11m42.523152572s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652177673 (469h11m42.523260708s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd): protocol: write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection; write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection Dec 25 04:40:42.652184456 (469h11m42.523267501s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652230243 (469h11m42.523313308s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq): protocol: write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection Dec 25 04:40:42.652240693 (469h11m42.523323778s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652268265 (469h11m42.52335131s) [seg_pos={0 0}, upload=411834]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 04:40:42.652354770 (469h11m42.523437815s) [upload=411834]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 04:40:42.652361693 (469h11m42.523444738s) [upload=411834]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 04:40:42.959704440 (469h11m42.830787485s) [upload=411834]: Done uploading segments... Dec 25 04:40:42.959728234 (469h11m42.83081127s) [upload=411834]: Upload finished. err: <nil> Duration: 6.051677745s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 5.656507481s trace: https://jaeger.dev.storj.io/trace/681cf842d2260d41 FAIL (2025-12-25 04:54:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, Piece ID: ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ) context deadline exceeded; piecestore: (Node ID: 1DT68m6J4PpDk3XUDhDaaqV46NLKVz4A23716JLFxEAh122GeR, Piece ID: WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA) context deadline exceeded; piecestore: (Node ID: 12wWYnhWHSqqAFgje4p9Cr7WYaJ6dJWsCqvrx8ZqVFoakjuFeka, Piece ID: GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q) context deadline exceeded ======LOGS====== nline: false Dec 25 04:40:39.039085560 (469h11m38.910168605s) [upload=411834]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 04:40:39.039095028 (469h11m38.910178073s) [upload=411834]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 04:40:39.297407613 (469h11m39.168490658s) [seg_pos={0 0}, upload=411834]: Begin upload segment... Dec 25 04:40:39.297471084 (469h11m39.168554119s) [seg_pos={0 0}, upload=411834]: Done begin upload segment. Dec 25 04:40:39.297511581 (469h11m39.168594616s) [noise=true , address=145.79.227.51:4411 , piece_id=WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.297663310 (469h11m39.168746345s) [noise=true , address=145.79.227.17:4970 , piece_id=6DPKJ762IUN4A7UEZXLDERCVDW24QEKGEHFETAYZOKWQPVN2QJYA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.298405574 (469h11m39.169488619s) [noise=true , address=145.79.227.2:4114 , piece_id=5ISXZKQSH4CLLTF3ILGAXHV6STMLRUSZP6UATU2677X2FUPLBCRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.298797961 (469h11m39.169881006s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299124392 (469h11m39.170207437s) [noise=true , address=45.91.116.5:4300 , piece_id=NTWC3RWYQQBBKPMXGUG44R2B5GWC2Z4LDFU3GTWOWCT3JIRIZOOQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299438650 (469h11m39.170521685s) [noise=true , address=145.79.227.5:4221 , piece_id=C4SXUVTG2TZ4EIRBGL4TLOLBMYOAZ4KLIQSZOXKEOWAJBQQJLWSA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299794709 (469h11m39.170877744s) [noise=true , address=145.79.227.37:4567 , piece_id=TZ4C57ISUI3CB3VOLTSM5W5QVDI7PVXV67PFJI3ZSVSXHTKPYPSA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.300142221 (469h11m39.171225256s) [noise=true , address=145.79.227.8:4331 , piece_id=ELLVNJFOVY5HSAHZDQYGQVO7OYHFZNFHCBJKHQFV5K6EV2M3JTRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.300506354 (469h11m39.171589389s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.301142795 (469h11m39.17222584s) [noise=true , address=145.79.227.36:4517 , piece_id=HBQU7CMAO2KJS6ODW2WHICBAWQSH6TQQPOQSENKDEISWCWFN2CMA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.301625835 (469h11m39.17270891s) [noise=true , address=145.79.227.45:4229 , piece_id=GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.302170964 (469h11m39.173254009s) [noise=true , address=145.79.227.3:4150 , piece_id=OHGA4EA33AVEDCJLB6QSUJLD3W2TV6YNN6FBVFFKTTSF2B6XDPRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.302629947 (469h11m39.173712982s) [noise=true , address=145.79.227.19:4079 , piece_id=ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303079704 (469h11m39.174162749s) [noise=true , address=145.79.227.15:4875 , piece_id=FUMLT2ODHUE3ETSNWVKBYLSJXYVCHUW5KAJJE5AECNL3IAQYYCLQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303499804 (469h11m39.174582849s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303921386 (469h11m39.175004432s) [noise=true , address=145.79.227.47:4265 , piece_id=TVUXNR6K35A3CWFDN4JRKM4Q6RGBC4LP2XKPXJFN2AJL32ESEBPA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:41.172453460 (469h11m41.043536495s) [upload=411834]: Next returned nil segment Dec 25 04:40:41.172473798 (469h11m41.043556833s) [upload=411834]: Waiting for error group managing segments... Dec 25 04:40:41.483361556 (469h11m41.354444601s) [noise=true , address=145.79.227.37:4567 , piece_id=TZ4C57ISUI3CB3VOLTSM5W5QVDI7PVXV67PFJI3ZSVSXHTKPYPSA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.484283170 (469h11m41.355366205s) [noise=true , address=145.79.227.5:4221 , piece_id=C4SXUVTG2TZ4EIRBGL4TLOLBMYOAZ4KLIQSZOXKEOWAJBQQJLWSA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.515393699 (469h11m41.386476744s) [noise=true , address=145.79.227.51:4411 , piece_id=WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.526179387 (469h11m41.397262422s) [noise=true , address=45.91.116.5:4300 , piece_id=NTWC3RWYQQBBKPMXGUG44R2B5GWC2Z4LDFU3GTWOWCT3JIRIZOOQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.642380892 (469h11m41.513463937s) [noise=true , address=145.79.227.36:4517 , piece_id=HBQU7CMAO2KJS6ODW2WHICBAWQSH6TQQPOQSENKDEISWCWFN2CMA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.662326662 (469h11m41.533409707s) [noise=true , address=145.79.227.3:4150 , piece_id=OHGA4EA33AVEDCJLB6QSUJLD3W2TV6YNN6FBVFFKTTSF2B6XDPRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.758852130 (469h11m41.629935175s) [noise=true , address=145.79.227.47:4265 , piece_id=TVUXNR6K35A3CWFDN4JRKM4Q6RGBC4LP2XKPXJFN2AJL32ESEBPA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.823449082 (469h11m41.694532117s) [noise=true , address=145.79.227.17:4970 , piece_id=6DPKJ762IUN4A7UEZXLDERCVDW24QEKGEHFETAYZOKWQPVN2QJYA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.952636616 (469h11m41.823719661s) [noise=true , address=145.79.227.19:4079 , piece_id=ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.025177123 (469h11m41.896260168s) [noise=true , address=145.79.227.45:4229 , piece_id=GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.222526274 (469h11m42.093609319s) [noise=true , address=145.79.227.8:4331 , piece_id=ELLVNJFOVY5HSAHZDQYGQVO7OYHFZNFHCBJKHQFV5K6EV2M3JTRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.250024374 (469h11m42.121107409s) [noise=true , address=145.79.227.2:4114 , piece_id=5ISXZKQSH4CLLTF3ILGAXHV6STMLRUSZP6UATU2677X2FUPLBCRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.651787921 (469h11m42.522870966s) [noise=true , address=145.79.227.15:4875 , piece_id=FUMLT2ODHUE3ETSNWVKBYLSJXYVCHUW5KAJJE5AECNL3IAQYYCLQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.651814982 (469h11m42.522898027s) [seg_pos={0 0}, upload=411834]: Segment reached optimal threshold of 13 pieces. Dec 25 04:40:42.652052093 (469h11m42.523135129s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: write tcp 188.166.225.170:36650->77.90.31.15:4697: use of closed network connection; storage node overloaded, request limit: 2000 Dec 25 04:40:42.652069537 (469h11m42.523152572s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652177673 (469h11m42.523260708s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd): protocol: write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection; write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection Dec 25 04:40:42.652184456 (469h11m42.523267501s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652230243 (469h11m42.523313308s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq): protocol: write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection Dec 25 04:40:42.652240693 (469h11m42.523323778s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652268265 (469h11m42.52335131s) [seg_pos={0 0}, upload=411834]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 04:40:42.652354770 (469h11m42.523437815s) [upload=411834]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 04:40:42.652361693 (469h11m42.523444738s) [upload=411834]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 04:40:42.959704440 (469h11m42.830787485s) [upload=411834]: Done uploading segments... Dec 25 04:40:42.959728234 (469h11m42.83081127s) [upload=411834]: Upload finished. err: <nil> Duration: 5.656534763s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 5.719627103s trace: https://jaeger.dev.storj.io/trace/681cf842d2260d41 FAIL (2025-12-25 04:53:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, Piece ID: ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ) context deadline exceeded; piecestore: (Node ID: 1DT68m6J4PpDk3XUDhDaaqV46NLKVz4A23716JLFxEAh122GeR, Piece ID: WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA) context deadline exceeded; piecestore: (Node ID: 12wWYnhWHSqqAFgje4p9Cr7WYaJ6dJWsCqvrx8ZqVFoakjuFeka, Piece ID: GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q) context deadline exceeded ======LOGS====== nline: false Dec 25 04:40:39.039085560 (469h11m38.910168605s) [upload=411834]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 04:40:39.039095028 (469h11m38.910178073s) [upload=411834]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 04:40:39.297407613 (469h11m39.168490658s) [seg_pos={0 0}, upload=411834]: Begin upload segment... Dec 25 04:40:39.297471084 (469h11m39.168554119s) [seg_pos={0 0}, upload=411834]: Done begin upload segment. Dec 25 04:40:39.297511581 (469h11m39.168594616s) [noise=true , address=145.79.227.51:4411 , piece_id=WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.297663310 (469h11m39.168746345s) [noise=true , address=145.79.227.17:4970 , piece_id=6DPKJ762IUN4A7UEZXLDERCVDW24QEKGEHFETAYZOKWQPVN2QJYA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.298405574 (469h11m39.169488619s) [noise=true , address=145.79.227.2:4114 , piece_id=5ISXZKQSH4CLLTF3ILGAXHV6STMLRUSZP6UATU2677X2FUPLBCRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.298797961 (469h11m39.169881006s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299124392 (469h11m39.170207437s) [noise=true , address=45.91.116.5:4300 , piece_id=NTWC3RWYQQBBKPMXGUG44R2B5GWC2Z4LDFU3GTWOWCT3JIRIZOOQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299438650 (469h11m39.170521685s) [noise=true , address=145.79.227.5:4221 , piece_id=C4SXUVTG2TZ4EIRBGL4TLOLBMYOAZ4KLIQSZOXKEOWAJBQQJLWSA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299794709 (469h11m39.170877744s) [noise=true , address=145.79.227.37:4567 , piece_id=TZ4C57ISUI3CB3VOLTSM5W5QVDI7PVXV67PFJI3ZSVSXHTKPYPSA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.300142221 (469h11m39.171225256s) [noise=true , address=145.79.227.8:4331 , piece_id=ELLVNJFOVY5HSAHZDQYGQVO7OYHFZNFHCBJKHQFV5K6EV2M3JTRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.300506354 (469h11m39.171589389s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.301142795 (469h11m39.17222584s) [noise=true , address=145.79.227.36:4517 , piece_id=HBQU7CMAO2KJS6ODW2WHICBAWQSH6TQQPOQSENKDEISWCWFN2CMA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.301625835 (469h11m39.17270891s) [noise=true , address=145.79.227.45:4229 , piece_id=GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.302170964 (469h11m39.173254009s) [noise=true , address=145.79.227.3:4150 , piece_id=OHGA4EA33AVEDCJLB6QSUJLD3W2TV6YNN6FBVFFKTTSF2B6XDPRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.302629947 (469h11m39.173712982s) [noise=true , address=145.79.227.19:4079 , piece_id=ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303079704 (469h11m39.174162749s) [noise=true , address=145.79.227.15:4875 , piece_id=FUMLT2ODHUE3ETSNWVKBYLSJXYVCHUW5KAJJE5AECNL3IAQYYCLQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303499804 (469h11m39.174582849s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303921386 (469h11m39.175004432s) [noise=true , address=145.79.227.47:4265 , piece_id=TVUXNR6K35A3CWFDN4JRKM4Q6RGBC4LP2XKPXJFN2AJL32ESEBPA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:41.172453460 (469h11m41.043536495s) [upload=411834]: Next returned nil segment Dec 25 04:40:41.172473798 (469h11m41.043556833s) [upload=411834]: Waiting for error group managing segments... Dec 25 04:40:41.483361556 (469h11m41.354444601s) [noise=true , address=145.79.227.37:4567 , piece_id=TZ4C57ISUI3CB3VOLTSM5W5QVDI7PVXV67PFJI3ZSVSXHTKPYPSA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.484283170 (469h11m41.355366205s) [noise=true , address=145.79.227.5:4221 , piece_id=C4SXUVTG2TZ4EIRBGL4TLOLBMYOAZ4KLIQSZOXKEOWAJBQQJLWSA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.515393699 (469h11m41.386476744s) [noise=true , address=145.79.227.51:4411 , piece_id=WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.526179387 (469h11m41.397262422s) [noise=true , address=45.91.116.5:4300 , piece_id=NTWC3RWYQQBBKPMXGUG44R2B5GWC2Z4LDFU3GTWOWCT3JIRIZOOQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.642380892 (469h11m41.513463937s) [noise=true , address=145.79.227.36:4517 , piece_id=HBQU7CMAO2KJS6ODW2WHICBAWQSH6TQQPOQSENKDEISWCWFN2CMA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.662326662 (469h11m41.533409707s) [noise=true , address=145.79.227.3:4150 , piece_id=OHGA4EA33AVEDCJLB6QSUJLD3W2TV6YNN6FBVFFKTTSF2B6XDPRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.758852130 (469h11m41.629935175s) [noise=true , address=145.79.227.47:4265 , piece_id=TVUXNR6K35A3CWFDN4JRKM4Q6RGBC4LP2XKPXJFN2AJL32ESEBPA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.823449082 (469h11m41.694532117s) [noise=true , address=145.79.227.17:4970 , piece_id=6DPKJ762IUN4A7UEZXLDERCVDW24QEKGEHFETAYZOKWQPVN2QJYA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.952636616 (469h11m41.823719661s) [noise=true , address=145.79.227.19:4079 , piece_id=ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.025177123 (469h11m41.896260168s) [noise=true , address=145.79.227.45:4229 , piece_id=GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.222526274 (469h11m42.093609319s) [noise=true , address=145.79.227.8:4331 , piece_id=ELLVNJFOVY5HSAHZDQYGQVO7OYHFZNFHCBJKHQFV5K6EV2M3JTRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.250024374 (469h11m42.121107409s) [noise=true , address=145.79.227.2:4114 , piece_id=5ISXZKQSH4CLLTF3ILGAXHV6STMLRUSZP6UATU2677X2FUPLBCRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.651787921 (469h11m42.522870966s) [noise=true , address=145.79.227.15:4875 , piece_id=FUMLT2ODHUE3ETSNWVKBYLSJXYVCHUW5KAJJE5AECNL3IAQYYCLQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.651814982 (469h11m42.522898027s) [seg_pos={0 0}, upload=411834]: Segment reached optimal threshold of 13 pieces. Dec 25 04:40:42.652052093 (469h11m42.523135129s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: write tcp 188.166.225.170:36650->77.90.31.15:4697: use of closed network connection; storage node overloaded, request limit: 2000 Dec 25 04:40:42.652069537 (469h11m42.523152572s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652177673 (469h11m42.523260708s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd): protocol: write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection; write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection Dec 25 04:40:42.652184456 (469h11m42.523267501s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652230243 (469h11m42.523313308s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq): protocol: write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection Dec 25 04:40:42.652240693 (469h11m42.523323778s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652268265 (469h11m42.52335131s) [seg_pos={0 0}, upload=411834]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 04:40:42.652354770 (469h11m42.523437815s) [upload=411834]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 04:40:42.652361693 (469h11m42.523444738s) [upload=411834]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 04:40:42.959704440 (469h11m42.830787485s) [upload=411834]: Done uploading segments... Dec 25 04:40:42.959728234 (469h11m42.83081127s) [upload=411834]: Upload finished. err: <nil> Duration: 5.719642602s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 5.976709406s trace: https://jaeger.dev.storj.io/trace/681cf842d2260d41 FAIL (2025-12-25 04:53:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, Piece ID: ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ) context deadline exceeded; piecestore: (Node ID: 1DT68m6J4PpDk3XUDhDaaqV46NLKVz4A23716JLFxEAh122GeR, Piece ID: WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA) context deadline exceeded; piecestore: (Node ID: 12wWYnhWHSqqAFgje4p9Cr7WYaJ6dJWsCqvrx8ZqVFoakjuFeka, Piece ID: GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q) context deadline exceeded ======LOGS====== nline: false Dec 25 04:40:39.039085560 (469h11m38.910168605s) [upload=411834]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 04:40:39.039095028 (469h11m38.910178073s) [upload=411834]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 04:40:39.297407613 (469h11m39.168490658s) [seg_pos={0 0}, upload=411834]: Begin upload segment... Dec 25 04:40:39.297471084 (469h11m39.168554119s) [seg_pos={0 0}, upload=411834]: Done begin upload segment. Dec 25 04:40:39.297511581 (469h11m39.168594616s) [noise=true , address=145.79.227.51:4411 , piece_id=WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.297663310 (469h11m39.168746345s) [noise=true , address=145.79.227.17:4970 , piece_id=6DPKJ762IUN4A7UEZXLDERCVDW24QEKGEHFETAYZOKWQPVN2QJYA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.298405574 (469h11m39.169488619s) [noise=true , address=145.79.227.2:4114 , piece_id=5ISXZKQSH4CLLTF3ILGAXHV6STMLRUSZP6UATU2677X2FUPLBCRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.298797961 (469h11m39.169881006s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299124392 (469h11m39.170207437s) [noise=true , address=45.91.116.5:4300 , piece_id=NTWC3RWYQQBBKPMXGUG44R2B5GWC2Z4LDFU3GTWOWCT3JIRIZOOQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299438650 (469h11m39.170521685s) [noise=true , address=145.79.227.5:4221 , piece_id=C4SXUVTG2TZ4EIRBGL4TLOLBMYOAZ4KLIQSZOXKEOWAJBQQJLWSA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299794709 (469h11m39.170877744s) [noise=true , address=145.79.227.37:4567 , piece_id=TZ4C57ISUI3CB3VOLTSM5W5QVDI7PVXV67PFJI3ZSVSXHTKPYPSA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.300142221 (469h11m39.171225256s) [noise=true , address=145.79.227.8:4331 , piece_id=ELLVNJFOVY5HSAHZDQYGQVO7OYHFZNFHCBJKHQFV5K6EV2M3JTRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.300506354 (469h11m39.171589389s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.301142795 (469h11m39.17222584s) [noise=true , address=145.79.227.36:4517 , piece_id=HBQU7CMAO2KJS6ODW2WHICBAWQSH6TQQPOQSENKDEISWCWFN2CMA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.301625835 (469h11m39.17270891s) [noise=true , address=145.79.227.45:4229 , piece_id=GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.302170964 (469h11m39.173254009s) [noise=true , address=145.79.227.3:4150 , piece_id=OHGA4EA33AVEDCJLB6QSUJLD3W2TV6YNN6FBVFFKTTSF2B6XDPRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.302629947 (469h11m39.173712982s) [noise=true , address=145.79.227.19:4079 , piece_id=ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303079704 (469h11m39.174162749s) [noise=true , address=145.79.227.15:4875 , piece_id=FUMLT2ODHUE3ETSNWVKBYLSJXYVCHUW5KAJJE5AECNL3IAQYYCLQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303499804 (469h11m39.174582849s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303921386 (469h11m39.175004432s) [noise=true , address=145.79.227.47:4265 , piece_id=TVUXNR6K35A3CWFDN4JRKM4Q6RGBC4LP2XKPXJFN2AJL32ESEBPA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:41.172453460 (469h11m41.043536495s) [upload=411834]: Next returned nil segment Dec 25 04:40:41.172473798 (469h11m41.043556833s) [upload=411834]: Waiting for error group managing segments... Dec 25 04:40:41.483361556 (469h11m41.354444601s) [noise=true , address=145.79.227.37:4567 , piece_id=TZ4C57ISUI3CB3VOLTSM5W5QVDI7PVXV67PFJI3ZSVSXHTKPYPSA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.484283170 (469h11m41.355366205s) [noise=true , address=145.79.227.5:4221 , piece_id=C4SXUVTG2TZ4EIRBGL4TLOLBMYOAZ4KLIQSZOXKEOWAJBQQJLWSA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.515393699 (469h11m41.386476744s) [noise=true , address=145.79.227.51:4411 , piece_id=WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.526179387 (469h11m41.397262422s) [noise=true , address=45.91.116.5:4300 , piece_id=NTWC3RWYQQBBKPMXGUG44R2B5GWC2Z4LDFU3GTWOWCT3JIRIZOOQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.642380892 (469h11m41.513463937s) [noise=true , address=145.79.227.36:4517 , piece_id=HBQU7CMAO2KJS6ODW2WHICBAWQSH6TQQPOQSENKDEISWCWFN2CMA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.662326662 (469h11m41.533409707s) [noise=true , address=145.79.227.3:4150 , piece_id=OHGA4EA33AVEDCJLB6QSUJLD3W2TV6YNN6FBVFFKTTSF2B6XDPRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.758852130 (469h11m41.629935175s) [noise=true , address=145.79.227.47:4265 , piece_id=TVUXNR6K35A3CWFDN4JRKM4Q6RGBC4LP2XKPXJFN2AJL32ESEBPA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.823449082 (469h11m41.694532117s) [noise=true , address=145.79.227.17:4970 , piece_id=6DPKJ762IUN4A7UEZXLDERCVDW24QEKGEHFETAYZOKWQPVN2QJYA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.952636616 (469h11m41.823719661s) [noise=true , address=145.79.227.19:4079 , piece_id=ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.025177123 (469h11m41.896260168s) [noise=true , address=145.79.227.45:4229 , piece_id=GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.222526274 (469h11m42.093609319s) [noise=true , address=145.79.227.8:4331 , piece_id=ELLVNJFOVY5HSAHZDQYGQVO7OYHFZNFHCBJKHQFV5K6EV2M3JTRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.250024374 (469h11m42.121107409s) [noise=true , address=145.79.227.2:4114 , piece_id=5ISXZKQSH4CLLTF3ILGAXHV6STMLRUSZP6UATU2677X2FUPLBCRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.651787921 (469h11m42.522870966s) [noise=true , address=145.79.227.15:4875 , piece_id=FUMLT2ODHUE3ETSNWVKBYLSJXYVCHUW5KAJJE5AECNL3IAQYYCLQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.651814982 (469h11m42.522898027s) [seg_pos={0 0}, upload=411834]: Segment reached optimal threshold of 13 pieces. Dec 25 04:40:42.652052093 (469h11m42.523135129s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: write tcp 188.166.225.170:36650->77.90.31.15:4697: use of closed network connection; storage node overloaded, request limit: 2000 Dec 25 04:40:42.652069537 (469h11m42.523152572s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652177673 (469h11m42.523260708s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd): protocol: write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection; write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection Dec 25 04:40:42.652184456 (469h11m42.523267501s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652230243 (469h11m42.523313308s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq): protocol: write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection Dec 25 04:40:42.652240693 (469h11m42.523323778s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652268265 (469h11m42.52335131s) [seg_pos={0 0}, upload=411834]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 04:40:42.652354770 (469h11m42.523437815s) [upload=411834]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 04:40:42.652361693 (469h11m42.523444738s) [upload=411834]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 04:40:42.959704440 (469h11m42.830787485s) [upload=411834]: Done uploading segments... Dec 25 04:40:42.959728234 (469h11m42.83081127s) [upload=411834]: Upload finished. err: <nil> Duration: 5.976768599s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 5.778695097s trace: https://jaeger.dev.storj.io/trace/681cf842d2260d41 FAIL (2025-12-25 04:53:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, Piece ID: ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ) context deadline exceeded; piecestore: (Node ID: 1DT68m6J4PpDk3XUDhDaaqV46NLKVz4A23716JLFxEAh122GeR, Piece ID: WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA) context deadline exceeded; piecestore: (Node ID: 12wWYnhWHSqqAFgje4p9Cr7WYaJ6dJWsCqvrx8ZqVFoakjuFeka, Piece ID: GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q) context deadline exceeded ======LOGS====== nline: false Dec 25 04:40:39.039085560 (469h11m38.910168605s) [upload=411834]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 04:40:39.039095028 (469h11m38.910178073s) [upload=411834]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 04:40:39.297407613 (469h11m39.168490658s) [seg_pos={0 0}, upload=411834]: Begin upload segment... Dec 25 04:40:39.297471084 (469h11m39.168554119s) [seg_pos={0 0}, upload=411834]: Done begin upload segment. Dec 25 04:40:39.297511581 (469h11m39.168594616s) [noise=true , address=145.79.227.51:4411 , piece_id=WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.297663310 (469h11m39.168746345s) [noise=true , address=145.79.227.17:4970 , piece_id=6DPKJ762IUN4A7UEZXLDERCVDW24QEKGEHFETAYZOKWQPVN2QJYA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.298405574 (469h11m39.169488619s) [noise=true , address=145.79.227.2:4114 , piece_id=5ISXZKQSH4CLLTF3ILGAXHV6STMLRUSZP6UATU2677X2FUPLBCRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.298797961 (469h11m39.169881006s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299124392 (469h11m39.170207437s) [noise=true , address=45.91.116.5:4300 , piece_id=NTWC3RWYQQBBKPMXGUG44R2B5GWC2Z4LDFU3GTWOWCT3JIRIZOOQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299438650 (469h11m39.170521685s) [noise=true , address=145.79.227.5:4221 , piece_id=C4SXUVTG2TZ4EIRBGL4TLOLBMYOAZ4KLIQSZOXKEOWAJBQQJLWSA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299794709 (469h11m39.170877744s) [noise=true , address=145.79.227.37:4567 , piece_id=TZ4C57ISUI3CB3VOLTSM5W5QVDI7PVXV67PFJI3ZSVSXHTKPYPSA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.300142221 (469h11m39.171225256s) [noise=true , address=145.79.227.8:4331 , piece_id=ELLVNJFOVY5HSAHZDQYGQVO7OYHFZNFHCBJKHQFV5K6EV2M3JTRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.300506354 (469h11m39.171589389s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.301142795 (469h11m39.17222584s) [noise=true , address=145.79.227.36:4517 , piece_id=HBQU7CMAO2KJS6ODW2WHICBAWQSH6TQQPOQSENKDEISWCWFN2CMA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.301625835 (469h11m39.17270891s) [noise=true , address=145.79.227.45:4229 , piece_id=GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.302170964 (469h11m39.173254009s) [noise=true , address=145.79.227.3:4150 , piece_id=OHGA4EA33AVEDCJLB6QSUJLD3W2TV6YNN6FBVFFKTTSF2B6XDPRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.302629947 (469h11m39.173712982s) [noise=true , address=145.79.227.19:4079 , piece_id=ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303079704 (469h11m39.174162749s) [noise=true , address=145.79.227.15:4875 , piece_id=FUMLT2ODHUE3ETSNWVKBYLSJXYVCHUW5KAJJE5AECNL3IAQYYCLQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303499804 (469h11m39.174582849s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303921386 (469h11m39.175004432s) [noise=true , address=145.79.227.47:4265 , piece_id=TVUXNR6K35A3CWFDN4JRKM4Q6RGBC4LP2XKPXJFN2AJL32ESEBPA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:41.172453460 (469h11m41.043536495s) [upload=411834]: Next returned nil segment Dec 25 04:40:41.172473798 (469h11m41.043556833s) [upload=411834]: Waiting for error group managing segments... Dec 25 04:40:41.483361556 (469h11m41.354444601s) [noise=true , address=145.79.227.37:4567 , piece_id=TZ4C57ISUI3CB3VOLTSM5W5QVDI7PVXV67PFJI3ZSVSXHTKPYPSA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.484283170 (469h11m41.355366205s) [noise=true , address=145.79.227.5:4221 , piece_id=C4SXUVTG2TZ4EIRBGL4TLOLBMYOAZ4KLIQSZOXKEOWAJBQQJLWSA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.515393699 (469h11m41.386476744s) [noise=true , address=145.79.227.51:4411 , piece_id=WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.526179387 (469h11m41.397262422s) [noise=true , address=45.91.116.5:4300 , piece_id=NTWC3RWYQQBBKPMXGUG44R2B5GWC2Z4LDFU3GTWOWCT3JIRIZOOQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.642380892 (469h11m41.513463937s) [noise=true , address=145.79.227.36:4517 , piece_id=HBQU7CMAO2KJS6ODW2WHICBAWQSH6TQQPOQSENKDEISWCWFN2CMA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.662326662 (469h11m41.533409707s) [noise=true , address=145.79.227.3:4150 , piece_id=OHGA4EA33AVEDCJLB6QSUJLD3W2TV6YNN6FBVFFKTTSF2B6XDPRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.758852130 (469h11m41.629935175s) [noise=true , address=145.79.227.47:4265 , piece_id=TVUXNR6K35A3CWFDN4JRKM4Q6RGBC4LP2XKPXJFN2AJL32ESEBPA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.823449082 (469h11m41.694532117s) [noise=true , address=145.79.227.17:4970 , piece_id=6DPKJ762IUN4A7UEZXLDERCVDW24QEKGEHFETAYZOKWQPVN2QJYA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.952636616 (469h11m41.823719661s) [noise=true , address=145.79.227.19:4079 , piece_id=ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.025177123 (469h11m41.896260168s) [noise=true , address=145.79.227.45:4229 , piece_id=GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.222526274 (469h11m42.093609319s) [noise=true , address=145.79.227.8:4331 , piece_id=ELLVNJFOVY5HSAHZDQYGQVO7OYHFZNFHCBJKHQFV5K6EV2M3JTRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.250024374 (469h11m42.121107409s) [noise=true , address=145.79.227.2:4114 , piece_id=5ISXZKQSH4CLLTF3ILGAXHV6STMLRUSZP6UATU2677X2FUPLBCRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.651787921 (469h11m42.522870966s) [noise=true , address=145.79.227.15:4875 , piece_id=FUMLT2ODHUE3ETSNWVKBYLSJXYVCHUW5KAJJE5AECNL3IAQYYCLQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.651814982 (469h11m42.522898027s) [seg_pos={0 0}, upload=411834]: Segment reached optimal threshold of 13 pieces. Dec 25 04:40:42.652052093 (469h11m42.523135129s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: write tcp 188.166.225.170:36650->77.90.31.15:4697: use of closed network connection; storage node overloaded, request limit: 2000 Dec 25 04:40:42.652069537 (469h11m42.523152572s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652177673 (469h11m42.523260708s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd): protocol: write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection; write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection Dec 25 04:40:42.652184456 (469h11m42.523267501s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652230243 (469h11m42.523313308s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq): protocol: write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection Dec 25 04:40:42.652240693 (469h11m42.523323778s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652268265 (469h11m42.52335131s) [seg_pos={0 0}, upload=411834]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 04:40:42.652354770 (469h11m42.523437815s) [upload=411834]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 04:40:42.652361693 (469h11m42.523444738s) [upload=411834]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 04:40:42.959704440 (469h11m42.830787485s) [upload=411834]: Done uploading segments... Dec 25 04:40:42.959728234 (469h11m42.83081127s) [upload=411834]: Upload finished. err: <nil> Duration: 5.778726736s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 5.723421045s trace: https://jaeger.dev.storj.io/trace/681cf842d2260d41 FAIL (2025-12-25 04:52:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, Piece ID: ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ) context deadline exceeded; piecestore: (Node ID: 1DT68m6J4PpDk3XUDhDaaqV46NLKVz4A23716JLFxEAh122GeR, Piece ID: WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA) context deadline exceeded; piecestore: (Node ID: 12wWYnhWHSqqAFgje4p9Cr7WYaJ6dJWsCqvrx8ZqVFoakjuFeka, Piece ID: GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q) context deadline exceeded ======LOGS====== nline: false Dec 25 04:40:39.039085560 (469h11m38.910168605s) [upload=411834]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 04:40:39.039095028 (469h11m38.910178073s) [upload=411834]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 04:40:39.297407613 (469h11m39.168490658s) [seg_pos={0 0}, upload=411834]: Begin upload segment... Dec 25 04:40:39.297471084 (469h11m39.168554119s) [seg_pos={0 0}, upload=411834]: Done begin upload segment. Dec 25 04:40:39.297511581 (469h11m39.168594616s) [noise=true , address=145.79.227.51:4411 , piece_id=WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.297663310 (469h11m39.168746345s) [noise=true , address=145.79.227.17:4970 , piece_id=6DPKJ762IUN4A7UEZXLDERCVDW24QEKGEHFETAYZOKWQPVN2QJYA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.298405574 (469h11m39.169488619s) [noise=true , address=145.79.227.2:4114 , piece_id=5ISXZKQSH4CLLTF3ILGAXHV6STMLRUSZP6UATU2677X2FUPLBCRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.298797961 (469h11m39.169881006s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299124392 (469h11m39.170207437s) [noise=true , address=45.91.116.5:4300 , piece_id=NTWC3RWYQQBBKPMXGUG44R2B5GWC2Z4LDFU3GTWOWCT3JIRIZOOQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299438650 (469h11m39.170521685s) [noise=true , address=145.79.227.5:4221 , piece_id=C4SXUVTG2TZ4EIRBGL4TLOLBMYOAZ4KLIQSZOXKEOWAJBQQJLWSA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299794709 (469h11m39.170877744s) [noise=true , address=145.79.227.37:4567 , piece_id=TZ4C57ISUI3CB3VOLTSM5W5QVDI7PVXV67PFJI3ZSVSXHTKPYPSA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.300142221 (469h11m39.171225256s) [noise=true , address=145.79.227.8:4331 , piece_id=ELLVNJFOVY5HSAHZDQYGQVO7OYHFZNFHCBJKHQFV5K6EV2M3JTRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.300506354 (469h11m39.171589389s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.301142795 (469h11m39.17222584s) [noise=true , address=145.79.227.36:4517 , piece_id=HBQU7CMAO2KJS6ODW2WHICBAWQSH6TQQPOQSENKDEISWCWFN2CMA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.301625835 (469h11m39.17270891s) [noise=true , address=145.79.227.45:4229 , piece_id=GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.302170964 (469h11m39.173254009s) [noise=true , address=145.79.227.3:4150 , piece_id=OHGA4EA33AVEDCJLB6QSUJLD3W2TV6YNN6FBVFFKTTSF2B6XDPRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.302629947 (469h11m39.173712982s) [noise=true , address=145.79.227.19:4079 , piece_id=ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303079704 (469h11m39.174162749s) [noise=true , address=145.79.227.15:4875 , piece_id=FUMLT2ODHUE3ETSNWVKBYLSJXYVCHUW5KAJJE5AECNL3IAQYYCLQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303499804 (469h11m39.174582849s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303921386 (469h11m39.175004432s) [noise=true , address=145.79.227.47:4265 , piece_id=TVUXNR6K35A3CWFDN4JRKM4Q6RGBC4LP2XKPXJFN2AJL32ESEBPA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:41.172453460 (469h11m41.043536495s) [upload=411834]: Next returned nil segment Dec 25 04:40:41.172473798 (469h11m41.043556833s) [upload=411834]: Waiting for error group managing segments... Dec 25 04:40:41.483361556 (469h11m41.354444601s) [noise=true , address=145.79.227.37:4567 , piece_id=TZ4C57ISUI3CB3VOLTSM5W5QVDI7PVXV67PFJI3ZSVSXHTKPYPSA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.484283170 (469h11m41.355366205s) [noise=true , address=145.79.227.5:4221 , piece_id=C4SXUVTG2TZ4EIRBGL4TLOLBMYOAZ4KLIQSZOXKEOWAJBQQJLWSA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.515393699 (469h11m41.386476744s) [noise=true , address=145.79.227.51:4411 , piece_id=WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.526179387 (469h11m41.397262422s) [noise=true , address=45.91.116.5:4300 , piece_id=NTWC3RWYQQBBKPMXGUG44R2B5GWC2Z4LDFU3GTWOWCT3JIRIZOOQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.642380892 (469h11m41.513463937s) [noise=true , address=145.79.227.36:4517 , piece_id=HBQU7CMAO2KJS6ODW2WHICBAWQSH6TQQPOQSENKDEISWCWFN2CMA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.662326662 (469h11m41.533409707s) [noise=true , address=145.79.227.3:4150 , piece_id=OHGA4EA33AVEDCJLB6QSUJLD3W2TV6YNN6FBVFFKTTSF2B6XDPRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.758852130 (469h11m41.629935175s) [noise=true , address=145.79.227.47:4265 , piece_id=TVUXNR6K35A3CWFDN4JRKM4Q6RGBC4LP2XKPXJFN2AJL32ESEBPA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.823449082 (469h11m41.694532117s) [noise=true , address=145.79.227.17:4970 , piece_id=6DPKJ762IUN4A7UEZXLDERCVDW24QEKGEHFETAYZOKWQPVN2QJYA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.952636616 (469h11m41.823719661s) [noise=true , address=145.79.227.19:4079 , piece_id=ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.025177123 (469h11m41.896260168s) [noise=true , address=145.79.227.45:4229 , piece_id=GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.222526274 (469h11m42.093609319s) [noise=true , address=145.79.227.8:4331 , piece_id=ELLVNJFOVY5HSAHZDQYGQVO7OYHFZNFHCBJKHQFV5K6EV2M3JTRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.250024374 (469h11m42.121107409s) [noise=true , address=145.79.227.2:4114 , piece_id=5ISXZKQSH4CLLTF3ILGAXHV6STMLRUSZP6UATU2677X2FUPLBCRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.651787921 (469h11m42.522870966s) [noise=true , address=145.79.227.15:4875 , piece_id=FUMLT2ODHUE3ETSNWVKBYLSJXYVCHUW5KAJJE5AECNL3IAQYYCLQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.651814982 (469h11m42.522898027s) [seg_pos={0 0}, upload=411834]: Segment reached optimal threshold of 13 pieces. Dec 25 04:40:42.652052093 (469h11m42.523135129s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: write tcp 188.166.225.170:36650->77.90.31.15:4697: use of closed network connection; storage node overloaded, request limit: 2000 Dec 25 04:40:42.652069537 (469h11m42.523152572s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652177673 (469h11m42.523260708s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd): protocol: write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection; write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection Dec 25 04:40:42.652184456 (469h11m42.523267501s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652230243 (469h11m42.523313308s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq): protocol: write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection Dec 25 04:40:42.652240693 (469h11m42.523323778s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652268265 (469h11m42.52335131s) [seg_pos={0 0}, upload=411834]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 04:40:42.652354770 (469h11m42.523437815s) [upload=411834]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 04:40:42.652361693 (469h11m42.523444738s) [upload=411834]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 04:40:42.959704440 (469h11m42.830787485s) [upload=411834]: Done uploading segments... Dec 25 04:40:42.959728234 (469h11m42.83081127s) [upload=411834]: Upload finished. err: <nil> Duration: 5.723450761s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 5.825723072s trace: https://jaeger.dev.storj.io/trace/681cf842d2260d41 FAIL (2025-12-25 04:51:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, Piece ID: ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ) context deadline exceeded; piecestore: (Node ID: 1DT68m6J4PpDk3XUDhDaaqV46NLKVz4A23716JLFxEAh122GeR, Piece ID: WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA) context deadline exceeded; piecestore: (Node ID: 12wWYnhWHSqqAFgje4p9Cr7WYaJ6dJWsCqvrx8ZqVFoakjuFeka, Piece ID: GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q) context deadline exceeded ======LOGS====== nline: false Dec 25 04:40:39.039085560 (469h11m38.910168605s) [upload=411834]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 04:40:39.039095028 (469h11m38.910178073s) [upload=411834]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 04:40:39.297407613 (469h11m39.168490658s) [seg_pos={0 0}, upload=411834]: Begin upload segment... Dec 25 04:40:39.297471084 (469h11m39.168554119s) [seg_pos={0 0}, upload=411834]: Done begin upload segment. Dec 25 04:40:39.297511581 (469h11m39.168594616s) [noise=true , address=145.79.227.51:4411 , piece_id=WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.297663310 (469h11m39.168746345s) [noise=true , address=145.79.227.17:4970 , piece_id=6DPKJ762IUN4A7UEZXLDERCVDW24QEKGEHFETAYZOKWQPVN2QJYA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.298405574 (469h11m39.169488619s) [noise=true , address=145.79.227.2:4114 , piece_id=5ISXZKQSH4CLLTF3ILGAXHV6STMLRUSZP6UATU2677X2FUPLBCRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.298797961 (469h11m39.169881006s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299124392 (469h11m39.170207437s) [noise=true , address=45.91.116.5:4300 , piece_id=NTWC3RWYQQBBKPMXGUG44R2B5GWC2Z4LDFU3GTWOWCT3JIRIZOOQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299438650 (469h11m39.170521685s) [noise=true , address=145.79.227.5:4221 , piece_id=C4SXUVTG2TZ4EIRBGL4TLOLBMYOAZ4KLIQSZOXKEOWAJBQQJLWSA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299794709 (469h11m39.170877744s) [noise=true , address=145.79.227.37:4567 , piece_id=TZ4C57ISUI3CB3VOLTSM5W5QVDI7PVXV67PFJI3ZSVSXHTKPYPSA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.300142221 (469h11m39.171225256s) [noise=true , address=145.79.227.8:4331 , piece_id=ELLVNJFOVY5HSAHZDQYGQVO7OYHFZNFHCBJKHQFV5K6EV2M3JTRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.300506354 (469h11m39.171589389s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.301142795 (469h11m39.17222584s) [noise=true , address=145.79.227.36:4517 , piece_id=HBQU7CMAO2KJS6ODW2WHICBAWQSH6TQQPOQSENKDEISWCWFN2CMA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.301625835 (469h11m39.17270891s) [noise=true , address=145.79.227.45:4229 , piece_id=GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.302170964 (469h11m39.173254009s) [noise=true , address=145.79.227.3:4150 , piece_id=OHGA4EA33AVEDCJLB6QSUJLD3W2TV6YNN6FBVFFKTTSF2B6XDPRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.302629947 (469h11m39.173712982s) [noise=true , address=145.79.227.19:4079 , piece_id=ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303079704 (469h11m39.174162749s) [noise=true , address=145.79.227.15:4875 , piece_id=FUMLT2ODHUE3ETSNWVKBYLSJXYVCHUW5KAJJE5AECNL3IAQYYCLQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303499804 (469h11m39.174582849s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303921386 (469h11m39.175004432s) [noise=true , address=145.79.227.47:4265 , piece_id=TVUXNR6K35A3CWFDN4JRKM4Q6RGBC4LP2XKPXJFN2AJL32ESEBPA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:41.172453460 (469h11m41.043536495s) [upload=411834]: Next returned nil segment Dec 25 04:40:41.172473798 (469h11m41.043556833s) [upload=411834]: Waiting for error group managing segments... Dec 25 04:40:41.483361556 (469h11m41.354444601s) [noise=true , address=145.79.227.37:4567 , piece_id=TZ4C57ISUI3CB3VOLTSM5W5QVDI7PVXV67PFJI3ZSVSXHTKPYPSA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.484283170 (469h11m41.355366205s) [noise=true , address=145.79.227.5:4221 , piece_id=C4SXUVTG2TZ4EIRBGL4TLOLBMYOAZ4KLIQSZOXKEOWAJBQQJLWSA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.515393699 (469h11m41.386476744s) [noise=true , address=145.79.227.51:4411 , piece_id=WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.526179387 (469h11m41.397262422s) [noise=true , address=45.91.116.5:4300 , piece_id=NTWC3RWYQQBBKPMXGUG44R2B5GWC2Z4LDFU3GTWOWCT3JIRIZOOQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.642380892 (469h11m41.513463937s) [noise=true , address=145.79.227.36:4517 , piece_id=HBQU7CMAO2KJS6ODW2WHICBAWQSH6TQQPOQSENKDEISWCWFN2CMA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.662326662 (469h11m41.533409707s) [noise=true , address=145.79.227.3:4150 , piece_id=OHGA4EA33AVEDCJLB6QSUJLD3W2TV6YNN6FBVFFKTTSF2B6XDPRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.758852130 (469h11m41.629935175s) [noise=true , address=145.79.227.47:4265 , piece_id=TVUXNR6K35A3CWFDN4JRKM4Q6RGBC4LP2XKPXJFN2AJL32ESEBPA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.823449082 (469h11m41.694532117s) [noise=true , address=145.79.227.17:4970 , piece_id=6DPKJ762IUN4A7UEZXLDERCVDW24QEKGEHFETAYZOKWQPVN2QJYA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.952636616 (469h11m41.823719661s) [noise=true , address=145.79.227.19:4079 , piece_id=ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.025177123 (469h11m41.896260168s) [noise=true , address=145.79.227.45:4229 , piece_id=GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.222526274 (469h11m42.093609319s) [noise=true , address=145.79.227.8:4331 , piece_id=ELLVNJFOVY5HSAHZDQYGQVO7OYHFZNFHCBJKHQFV5K6EV2M3JTRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.250024374 (469h11m42.121107409s) [noise=true , address=145.79.227.2:4114 , piece_id=5ISXZKQSH4CLLTF3ILGAXHV6STMLRUSZP6UATU2677X2FUPLBCRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.651787921 (469h11m42.522870966s) [noise=true , address=145.79.227.15:4875 , piece_id=FUMLT2ODHUE3ETSNWVKBYLSJXYVCHUW5KAJJE5AECNL3IAQYYCLQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.651814982 (469h11m42.522898027s) [seg_pos={0 0}, upload=411834]: Segment reached optimal threshold of 13 pieces. Dec 25 04:40:42.652052093 (469h11m42.523135129s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: write tcp 188.166.225.170:36650->77.90.31.15:4697: use of closed network connection; storage node overloaded, request limit: 2000 Dec 25 04:40:42.652069537 (469h11m42.523152572s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652177673 (469h11m42.523260708s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd): protocol: write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection; write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection Dec 25 04:40:42.652184456 (469h11m42.523267501s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652230243 (469h11m42.523313308s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq): protocol: write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection Dec 25 04:40:42.652240693 (469h11m42.523323778s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652268265 (469h11m42.52335131s) [seg_pos={0 0}, upload=411834]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 04:40:42.652354770 (469h11m42.523437815s) [upload=411834]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 04:40:42.652361693 (469h11m42.523444738s) [upload=411834]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 04:40:42.959704440 (469h11m42.830787485s) [upload=411834]: Done uploading segments... Dec 25 04:40:42.959728234 (469h11m42.83081127s) [upload=411834]: Upload finished. err: <nil> Duration: 5.825755744s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 5.702438452s trace: https://jaeger.dev.storj.io/trace/681cf842d2260d41 FAIL (2025-12-25 04:50:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, Piece ID: ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ) context deadline exceeded; piecestore: (Node ID: 1DT68m6J4PpDk3XUDhDaaqV46NLKVz4A23716JLFxEAh122GeR, Piece ID: WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA) context deadline exceeded; piecestore: (Node ID: 12wWYnhWHSqqAFgje4p9Cr7WYaJ6dJWsCqvrx8ZqVFoakjuFeka, Piece ID: GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q) context deadline exceeded ======LOGS====== nline: false Dec 25 04:40:39.039085560 (469h11m38.910168605s) [upload=411834]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 04:40:39.039095028 (469h11m38.910178073s) [upload=411834]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 04:40:39.297407613 (469h11m39.168490658s) [seg_pos={0 0}, upload=411834]: Begin upload segment... Dec 25 04:40:39.297471084 (469h11m39.168554119s) [seg_pos={0 0}, upload=411834]: Done begin upload segment. Dec 25 04:40:39.297511581 (469h11m39.168594616s) [noise=true , address=145.79.227.51:4411 , piece_id=WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.297663310 (469h11m39.168746345s) [noise=true , address=145.79.227.17:4970 , piece_id=6DPKJ762IUN4A7UEZXLDERCVDW24QEKGEHFETAYZOKWQPVN2QJYA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.298405574 (469h11m39.169488619s) [noise=true , address=145.79.227.2:4114 , piece_id=5ISXZKQSH4CLLTF3ILGAXHV6STMLRUSZP6UATU2677X2FUPLBCRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.298797961 (469h11m39.169881006s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299124392 (469h11m39.170207437s) [noise=true , address=45.91.116.5:4300 , piece_id=NTWC3RWYQQBBKPMXGUG44R2B5GWC2Z4LDFU3GTWOWCT3JIRIZOOQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299438650 (469h11m39.170521685s) [noise=true , address=145.79.227.5:4221 , piece_id=C4SXUVTG2TZ4EIRBGL4TLOLBMYOAZ4KLIQSZOXKEOWAJBQQJLWSA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299794709 (469h11m39.170877744s) [noise=true , address=145.79.227.37:4567 , piece_id=TZ4C57ISUI3CB3VOLTSM5W5QVDI7PVXV67PFJI3ZSVSXHTKPYPSA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.300142221 (469h11m39.171225256s) [noise=true , address=145.79.227.8:4331 , piece_id=ELLVNJFOVY5HSAHZDQYGQVO7OYHFZNFHCBJKHQFV5K6EV2M3JTRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.300506354 (469h11m39.171589389s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.301142795 (469h11m39.17222584s) [noise=true , address=145.79.227.36:4517 , piece_id=HBQU7CMAO2KJS6ODW2WHICBAWQSH6TQQPOQSENKDEISWCWFN2CMA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.301625835 (469h11m39.17270891s) [noise=true , address=145.79.227.45:4229 , piece_id=GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.302170964 (469h11m39.173254009s) [noise=true , address=145.79.227.3:4150 , piece_id=OHGA4EA33AVEDCJLB6QSUJLD3W2TV6YNN6FBVFFKTTSF2B6XDPRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.302629947 (469h11m39.173712982s) [noise=true , address=145.79.227.19:4079 , piece_id=ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303079704 (469h11m39.174162749s) [noise=true , address=145.79.227.15:4875 , piece_id=FUMLT2ODHUE3ETSNWVKBYLSJXYVCHUW5KAJJE5AECNL3IAQYYCLQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303499804 (469h11m39.174582849s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303921386 (469h11m39.175004432s) [noise=true , address=145.79.227.47:4265 , piece_id=TVUXNR6K35A3CWFDN4JRKM4Q6RGBC4LP2XKPXJFN2AJL32ESEBPA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:41.172453460 (469h11m41.043536495s) [upload=411834]: Next returned nil segment Dec 25 04:40:41.172473798 (469h11m41.043556833s) [upload=411834]: Waiting for error group managing segments... Dec 25 04:40:41.483361556 (469h11m41.354444601s) [noise=true , address=145.79.227.37:4567 , piece_id=TZ4C57ISUI3CB3VOLTSM5W5QVDI7PVXV67PFJI3ZSVSXHTKPYPSA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.484283170 (469h11m41.355366205s) [noise=true , address=145.79.227.5:4221 , piece_id=C4SXUVTG2TZ4EIRBGL4TLOLBMYOAZ4KLIQSZOXKEOWAJBQQJLWSA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.515393699 (469h11m41.386476744s) [noise=true , address=145.79.227.51:4411 , piece_id=WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.526179387 (469h11m41.397262422s) [noise=true , address=45.91.116.5:4300 , piece_id=NTWC3RWYQQBBKPMXGUG44R2B5GWC2Z4LDFU3GTWOWCT3JIRIZOOQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.642380892 (469h11m41.513463937s) [noise=true , address=145.79.227.36:4517 , piece_id=HBQU7CMAO2KJS6ODW2WHICBAWQSH6TQQPOQSENKDEISWCWFN2CMA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.662326662 (469h11m41.533409707s) [noise=true , address=145.79.227.3:4150 , piece_id=OHGA4EA33AVEDCJLB6QSUJLD3W2TV6YNN6FBVFFKTTSF2B6XDPRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.758852130 (469h11m41.629935175s) [noise=true , address=145.79.227.47:4265 , piece_id=TVUXNR6K35A3CWFDN4JRKM4Q6RGBC4LP2XKPXJFN2AJL32ESEBPA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.823449082 (469h11m41.694532117s) [noise=true , address=145.79.227.17:4970 , piece_id=6DPKJ762IUN4A7UEZXLDERCVDW24QEKGEHFETAYZOKWQPVN2QJYA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.952636616 (469h11m41.823719661s) [noise=true , address=145.79.227.19:4079 , piece_id=ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.025177123 (469h11m41.896260168s) [noise=true , address=145.79.227.45:4229 , piece_id=GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.222526274 (469h11m42.093609319s) [noise=true , address=145.79.227.8:4331 , piece_id=ELLVNJFOVY5HSAHZDQYGQVO7OYHFZNFHCBJKHQFV5K6EV2M3JTRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.250024374 (469h11m42.121107409s) [noise=true , address=145.79.227.2:4114 , piece_id=5ISXZKQSH4CLLTF3ILGAXHV6STMLRUSZP6UATU2677X2FUPLBCRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.651787921 (469h11m42.522870966s) [noise=true , address=145.79.227.15:4875 , piece_id=FUMLT2ODHUE3ETSNWVKBYLSJXYVCHUW5KAJJE5AECNL3IAQYYCLQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.651814982 (469h11m42.522898027s) [seg_pos={0 0}, upload=411834]: Segment reached optimal threshold of 13 pieces. Dec 25 04:40:42.652052093 (469h11m42.523135129s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: write tcp 188.166.225.170:36650->77.90.31.15:4697: use of closed network connection; storage node overloaded, request limit: 2000 Dec 25 04:40:42.652069537 (469h11m42.523152572s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652177673 (469h11m42.523260708s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd): protocol: write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection; write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection Dec 25 04:40:42.652184456 (469h11m42.523267501s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652230243 (469h11m42.523313308s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq): protocol: write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection Dec 25 04:40:42.652240693 (469h11m42.523323778s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652268265 (469h11m42.52335131s) [seg_pos={0 0}, upload=411834]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 04:40:42.652354770 (469h11m42.523437815s) [upload=411834]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 04:40:42.652361693 (469h11m42.523444738s) [upload=411834]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 04:40:42.959704440 (469h11m42.830787485s) [upload=411834]: Done uploading segments... Dec 25 04:40:42.959728234 (469h11m42.83081127s) [upload=411834]: Upload finished. err: <nil> Duration: 5.702470343s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 5.659506613s trace: https://jaeger.dev.storj.io/trace/681cf842d2260d41 FAIL (2025-12-25 04:49:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, Piece ID: ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ) context deadline exceeded; piecestore: (Node ID: 1DT68m6J4PpDk3XUDhDaaqV46NLKVz4A23716JLFxEAh122GeR, Piece ID: WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA) context deadline exceeded; piecestore: (Node ID: 12wWYnhWHSqqAFgje4p9Cr7WYaJ6dJWsCqvrx8ZqVFoakjuFeka, Piece ID: GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q) context deadline exceeded ======LOGS====== nline: false Dec 25 04:40:39.039085560 (469h11m38.910168605s) [upload=411834]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 04:40:39.039095028 (469h11m38.910178073s) [upload=411834]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 04:40:39.297407613 (469h11m39.168490658s) [seg_pos={0 0}, upload=411834]: Begin upload segment... Dec 25 04:40:39.297471084 (469h11m39.168554119s) [seg_pos={0 0}, upload=411834]: Done begin upload segment. Dec 25 04:40:39.297511581 (469h11m39.168594616s) [noise=true , address=145.79.227.51:4411 , piece_id=WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.297663310 (469h11m39.168746345s) [noise=true , address=145.79.227.17:4970 , piece_id=6DPKJ762IUN4A7UEZXLDERCVDW24QEKGEHFETAYZOKWQPVN2QJYA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.298405574 (469h11m39.169488619s) [noise=true , address=145.79.227.2:4114 , piece_id=5ISXZKQSH4CLLTF3ILGAXHV6STMLRUSZP6UATU2677X2FUPLBCRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.298797961 (469h11m39.169881006s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299124392 (469h11m39.170207437s) [noise=true , address=45.91.116.5:4300 , piece_id=NTWC3RWYQQBBKPMXGUG44R2B5GWC2Z4LDFU3GTWOWCT3JIRIZOOQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299438650 (469h11m39.170521685s) [noise=true , address=145.79.227.5:4221 , piece_id=C4SXUVTG2TZ4EIRBGL4TLOLBMYOAZ4KLIQSZOXKEOWAJBQQJLWSA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299794709 (469h11m39.170877744s) [noise=true , address=145.79.227.37:4567 , piece_id=TZ4C57ISUI3CB3VOLTSM5W5QVDI7PVXV67PFJI3ZSVSXHTKPYPSA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.300142221 (469h11m39.171225256s) [noise=true , address=145.79.227.8:4331 , piece_id=ELLVNJFOVY5HSAHZDQYGQVO7OYHFZNFHCBJKHQFV5K6EV2M3JTRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.300506354 (469h11m39.171589389s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.301142795 (469h11m39.17222584s) [noise=true , address=145.79.227.36:4517 , piece_id=HBQU7CMAO2KJS6ODW2WHICBAWQSH6TQQPOQSENKDEISWCWFN2CMA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.301625835 (469h11m39.17270891s) [noise=true , address=145.79.227.45:4229 , piece_id=GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.302170964 (469h11m39.173254009s) [noise=true , address=145.79.227.3:4150 , piece_id=OHGA4EA33AVEDCJLB6QSUJLD3W2TV6YNN6FBVFFKTTSF2B6XDPRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.302629947 (469h11m39.173712982s) [noise=true , address=145.79.227.19:4079 , piece_id=ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303079704 (469h11m39.174162749s) [noise=true , address=145.79.227.15:4875 , piece_id=FUMLT2ODHUE3ETSNWVKBYLSJXYVCHUW5KAJJE5AECNL3IAQYYCLQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303499804 (469h11m39.174582849s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303921386 (469h11m39.175004432s) [noise=true , address=145.79.227.47:4265 , piece_id=TVUXNR6K35A3CWFDN4JRKM4Q6RGBC4LP2XKPXJFN2AJL32ESEBPA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:41.172453460 (469h11m41.043536495s) [upload=411834]: Next returned nil segment Dec 25 04:40:41.172473798 (469h11m41.043556833s) [upload=411834]: Waiting for error group managing segments... Dec 25 04:40:41.483361556 (469h11m41.354444601s) [noise=true , address=145.79.227.37:4567 , piece_id=TZ4C57ISUI3CB3VOLTSM5W5QVDI7PVXV67PFJI3ZSVSXHTKPYPSA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.484283170 (469h11m41.355366205s) [noise=true , address=145.79.227.5:4221 , piece_id=C4SXUVTG2TZ4EIRBGL4TLOLBMYOAZ4KLIQSZOXKEOWAJBQQJLWSA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.515393699 (469h11m41.386476744s) [noise=true , address=145.79.227.51:4411 , piece_id=WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.526179387 (469h11m41.397262422s) [noise=true , address=45.91.116.5:4300 , piece_id=NTWC3RWYQQBBKPMXGUG44R2B5GWC2Z4LDFU3GTWOWCT3JIRIZOOQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.642380892 (469h11m41.513463937s) [noise=true , address=145.79.227.36:4517 , piece_id=HBQU7CMAO2KJS6ODW2WHICBAWQSH6TQQPOQSENKDEISWCWFN2CMA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.662326662 (469h11m41.533409707s) [noise=true , address=145.79.227.3:4150 , piece_id=OHGA4EA33AVEDCJLB6QSUJLD3W2TV6YNN6FBVFFKTTSF2B6XDPRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.758852130 (469h11m41.629935175s) [noise=true , address=145.79.227.47:4265 , piece_id=TVUXNR6K35A3CWFDN4JRKM4Q6RGBC4LP2XKPXJFN2AJL32ESEBPA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.823449082 (469h11m41.694532117s) [noise=true , address=145.79.227.17:4970 , piece_id=6DPKJ762IUN4A7UEZXLDERCVDW24QEKGEHFETAYZOKWQPVN2QJYA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.952636616 (469h11m41.823719661s) [noise=true , address=145.79.227.19:4079 , piece_id=ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.025177123 (469h11m41.896260168s) [noise=true , address=145.79.227.45:4229 , piece_id=GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.222526274 (469h11m42.093609319s) [noise=true , address=145.79.227.8:4331 , piece_id=ELLVNJFOVY5HSAHZDQYGQVO7OYHFZNFHCBJKHQFV5K6EV2M3JTRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.250024374 (469h11m42.121107409s) [noise=true , address=145.79.227.2:4114 , piece_id=5ISXZKQSH4CLLTF3ILGAXHV6STMLRUSZP6UATU2677X2FUPLBCRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.651787921 (469h11m42.522870966s) [noise=true , address=145.79.227.15:4875 , piece_id=FUMLT2ODHUE3ETSNWVKBYLSJXYVCHUW5KAJJE5AECNL3IAQYYCLQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.651814982 (469h11m42.522898027s) [seg_pos={0 0}, upload=411834]: Segment reached optimal threshold of 13 pieces. Dec 25 04:40:42.652052093 (469h11m42.523135129s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: write tcp 188.166.225.170:36650->77.90.31.15:4697: use of closed network connection; storage node overloaded, request limit: 2000 Dec 25 04:40:42.652069537 (469h11m42.523152572s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652177673 (469h11m42.523260708s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd): protocol: write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection; write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection Dec 25 04:40:42.652184456 (469h11m42.523267501s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652230243 (469h11m42.523313308s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq): protocol: write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection Dec 25 04:40:42.652240693 (469h11m42.523323778s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652268265 (469h11m42.52335131s) [seg_pos={0 0}, upload=411834]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 04:40:42.652354770 (469h11m42.523437815s) [upload=411834]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 04:40:42.652361693 (469h11m42.523444738s) [upload=411834]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 04:40:42.959704440 (469h11m42.830787485s) [upload=411834]: Done uploading segments... Dec 25 04:40:42.959728234 (469h11m42.83081127s) [upload=411834]: Upload finished. err: <nil> Duration: 5.659529806s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 5.664894496s trace: https://jaeger.dev.storj.io/trace/681cf842d2260d41 FAIL (2025-12-25 04:49:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, Piece ID: ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ) context deadline exceeded; piecestore: (Node ID: 1DT68m6J4PpDk3XUDhDaaqV46NLKVz4A23716JLFxEAh122GeR, Piece ID: WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA) context deadline exceeded; piecestore: (Node ID: 12wWYnhWHSqqAFgje4p9Cr7WYaJ6dJWsCqvrx8ZqVFoakjuFeka, Piece ID: GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q) context deadline exceeded ======LOGS====== nline: false Dec 25 04:40:39.039085560 (469h11m38.910168605s) [upload=411834]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 04:40:39.039095028 (469h11m38.910178073s) [upload=411834]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 04:40:39.297407613 (469h11m39.168490658s) [seg_pos={0 0}, upload=411834]: Begin upload segment... Dec 25 04:40:39.297471084 (469h11m39.168554119s) [seg_pos={0 0}, upload=411834]: Done begin upload segment. Dec 25 04:40:39.297511581 (469h11m39.168594616s) [noise=true , address=145.79.227.51:4411 , piece_id=WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.297663310 (469h11m39.168746345s) [noise=true , address=145.79.227.17:4970 , piece_id=6DPKJ762IUN4A7UEZXLDERCVDW24QEKGEHFETAYZOKWQPVN2QJYA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.298405574 (469h11m39.169488619s) [noise=true , address=145.79.227.2:4114 , piece_id=5ISXZKQSH4CLLTF3ILGAXHV6STMLRUSZP6UATU2677X2FUPLBCRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.298797961 (469h11m39.169881006s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299124392 (469h11m39.170207437s) [noise=true , address=45.91.116.5:4300 , piece_id=NTWC3RWYQQBBKPMXGUG44R2B5GWC2Z4LDFU3GTWOWCT3JIRIZOOQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299438650 (469h11m39.170521685s) [noise=true , address=145.79.227.5:4221 , piece_id=C4SXUVTG2TZ4EIRBGL4TLOLBMYOAZ4KLIQSZOXKEOWAJBQQJLWSA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299794709 (469h11m39.170877744s) [noise=true , address=145.79.227.37:4567 , piece_id=TZ4C57ISUI3CB3VOLTSM5W5QVDI7PVXV67PFJI3ZSVSXHTKPYPSA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.300142221 (469h11m39.171225256s) [noise=true , address=145.79.227.8:4331 , piece_id=ELLVNJFOVY5HSAHZDQYGQVO7OYHFZNFHCBJKHQFV5K6EV2M3JTRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.300506354 (469h11m39.171589389s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.301142795 (469h11m39.17222584s) [noise=true , address=145.79.227.36:4517 , piece_id=HBQU7CMAO2KJS6ODW2WHICBAWQSH6TQQPOQSENKDEISWCWFN2CMA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.301625835 (469h11m39.17270891s) [noise=true , address=145.79.227.45:4229 , piece_id=GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.302170964 (469h11m39.173254009s) [noise=true , address=145.79.227.3:4150 , piece_id=OHGA4EA33AVEDCJLB6QSUJLD3W2TV6YNN6FBVFFKTTSF2B6XDPRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.302629947 (469h11m39.173712982s) [noise=true , address=145.79.227.19:4079 , piece_id=ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303079704 (469h11m39.174162749s) [noise=true , address=145.79.227.15:4875 , piece_id=FUMLT2ODHUE3ETSNWVKBYLSJXYVCHUW5KAJJE5AECNL3IAQYYCLQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303499804 (469h11m39.174582849s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303921386 (469h11m39.175004432s) [noise=true , address=145.79.227.47:4265 , piece_id=TVUXNR6K35A3CWFDN4JRKM4Q6RGBC4LP2XKPXJFN2AJL32ESEBPA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:41.172453460 (469h11m41.043536495s) [upload=411834]: Next returned nil segment Dec 25 04:40:41.172473798 (469h11m41.043556833s) [upload=411834]: Waiting for error group managing segments... Dec 25 04:40:41.483361556 (469h11m41.354444601s) [noise=true , address=145.79.227.37:4567 , piece_id=TZ4C57ISUI3CB3VOLTSM5W5QVDI7PVXV67PFJI3ZSVSXHTKPYPSA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.484283170 (469h11m41.355366205s) [noise=true , address=145.79.227.5:4221 , piece_id=C4SXUVTG2TZ4EIRBGL4TLOLBMYOAZ4KLIQSZOXKEOWAJBQQJLWSA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.515393699 (469h11m41.386476744s) [noise=true , address=145.79.227.51:4411 , piece_id=WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.526179387 (469h11m41.397262422s) [noise=true , address=45.91.116.5:4300 , piece_id=NTWC3RWYQQBBKPMXGUG44R2B5GWC2Z4LDFU3GTWOWCT3JIRIZOOQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.642380892 (469h11m41.513463937s) [noise=true , address=145.79.227.36:4517 , piece_id=HBQU7CMAO2KJS6ODW2WHICBAWQSH6TQQPOQSENKDEISWCWFN2CMA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.662326662 (469h11m41.533409707s) [noise=true , address=145.79.227.3:4150 , piece_id=OHGA4EA33AVEDCJLB6QSUJLD3W2TV6YNN6FBVFFKTTSF2B6XDPRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.758852130 (469h11m41.629935175s) [noise=true , address=145.79.227.47:4265 , piece_id=TVUXNR6K35A3CWFDN4JRKM4Q6RGBC4LP2XKPXJFN2AJL32ESEBPA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.823449082 (469h11m41.694532117s) [noise=true , address=145.79.227.17:4970 , piece_id=6DPKJ762IUN4A7UEZXLDERCVDW24QEKGEHFETAYZOKWQPVN2QJYA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.952636616 (469h11m41.823719661s) [noise=true , address=145.79.227.19:4079 , piece_id=ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.025177123 (469h11m41.896260168s) [noise=true , address=145.79.227.45:4229 , piece_id=GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.222526274 (469h11m42.093609319s) [noise=true , address=145.79.227.8:4331 , piece_id=ELLVNJFOVY5HSAHZDQYGQVO7OYHFZNFHCBJKHQFV5K6EV2M3JTRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.250024374 (469h11m42.121107409s) [noise=true , address=145.79.227.2:4114 , piece_id=5ISXZKQSH4CLLTF3ILGAXHV6STMLRUSZP6UATU2677X2FUPLBCRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.651787921 (469h11m42.522870966s) [noise=true , address=145.79.227.15:4875 , piece_id=FUMLT2ODHUE3ETSNWVKBYLSJXYVCHUW5KAJJE5AECNL3IAQYYCLQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.651814982 (469h11m42.522898027s) [seg_pos={0 0}, upload=411834]: Segment reached optimal threshold of 13 pieces. Dec 25 04:40:42.652052093 (469h11m42.523135129s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: write tcp 188.166.225.170:36650->77.90.31.15:4697: use of closed network connection; storage node overloaded, request limit: 2000 Dec 25 04:40:42.652069537 (469h11m42.523152572s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652177673 (469h11m42.523260708s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd): protocol: write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection; write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection Dec 25 04:40:42.652184456 (469h11m42.523267501s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652230243 (469h11m42.523313308s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq): protocol: write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection Dec 25 04:40:42.652240693 (469h11m42.523323778s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652268265 (469h11m42.52335131s) [seg_pos={0 0}, upload=411834]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 04:40:42.652354770 (469h11m42.523437815s) [upload=411834]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 04:40:42.652361693 (469h11m42.523444738s) [upload=411834]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 04:40:42.959704440 (469h11m42.830787485s) [upload=411834]: Done uploading segments... Dec 25 04:40:42.959728234 (469h11m42.83081127s) [upload=411834]: Upload finished. err: <nil> Duration: 5.664916579s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 5.878649088s trace: https://jaeger.dev.storj.io/trace/681cf842d2260d41 FAIL (2025-12-25 04:48:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, Piece ID: ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ) context deadline exceeded; piecestore: (Node ID: 1DT68m6J4PpDk3XUDhDaaqV46NLKVz4A23716JLFxEAh122GeR, Piece ID: WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA) context deadline exceeded; piecestore: (Node ID: 12wWYnhWHSqqAFgje4p9Cr7WYaJ6dJWsCqvrx8ZqVFoakjuFeka, Piece ID: GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q) context deadline exceeded ======LOGS====== nline: false Dec 25 04:40:39.039085560 (469h11m38.910168605s) [upload=411834]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 04:40:39.039095028 (469h11m38.910178073s) [upload=411834]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 04:40:39.297407613 (469h11m39.168490658s) [seg_pos={0 0}, upload=411834]: Begin upload segment... Dec 25 04:40:39.297471084 (469h11m39.168554119s) [seg_pos={0 0}, upload=411834]: Done begin upload segment. Dec 25 04:40:39.297511581 (469h11m39.168594616s) [noise=true , address=145.79.227.51:4411 , piece_id=WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.297663310 (469h11m39.168746345s) [noise=true , address=145.79.227.17:4970 , piece_id=6DPKJ762IUN4A7UEZXLDERCVDW24QEKGEHFETAYZOKWQPVN2QJYA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.298405574 (469h11m39.169488619s) [noise=true , address=145.79.227.2:4114 , piece_id=5ISXZKQSH4CLLTF3ILGAXHV6STMLRUSZP6UATU2677X2FUPLBCRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.298797961 (469h11m39.169881006s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299124392 (469h11m39.170207437s) [noise=true , address=45.91.116.5:4300 , piece_id=NTWC3RWYQQBBKPMXGUG44R2B5GWC2Z4LDFU3GTWOWCT3JIRIZOOQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299438650 (469h11m39.170521685s) [noise=true , address=145.79.227.5:4221 , piece_id=C4SXUVTG2TZ4EIRBGL4TLOLBMYOAZ4KLIQSZOXKEOWAJBQQJLWSA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299794709 (469h11m39.170877744s) [noise=true , address=145.79.227.37:4567 , piece_id=TZ4C57ISUI3CB3VOLTSM5W5QVDI7PVXV67PFJI3ZSVSXHTKPYPSA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.300142221 (469h11m39.171225256s) [noise=true , address=145.79.227.8:4331 , piece_id=ELLVNJFOVY5HSAHZDQYGQVO7OYHFZNFHCBJKHQFV5K6EV2M3JTRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.300506354 (469h11m39.171589389s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.301142795 (469h11m39.17222584s) [noise=true , address=145.79.227.36:4517 , piece_id=HBQU7CMAO2KJS6ODW2WHICBAWQSH6TQQPOQSENKDEISWCWFN2CMA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.301625835 (469h11m39.17270891s) [noise=true , address=145.79.227.45:4229 , piece_id=GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.302170964 (469h11m39.173254009s) [noise=true , address=145.79.227.3:4150 , piece_id=OHGA4EA33AVEDCJLB6QSUJLD3W2TV6YNN6FBVFFKTTSF2B6XDPRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.302629947 (469h11m39.173712982s) [noise=true , address=145.79.227.19:4079 , piece_id=ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303079704 (469h11m39.174162749s) [noise=true , address=145.79.227.15:4875 , piece_id=FUMLT2ODHUE3ETSNWVKBYLSJXYVCHUW5KAJJE5AECNL3IAQYYCLQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303499804 (469h11m39.174582849s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303921386 (469h11m39.175004432s) [noise=true , address=145.79.227.47:4265 , piece_id=TVUXNR6K35A3CWFDN4JRKM4Q6RGBC4LP2XKPXJFN2AJL32ESEBPA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:41.172453460 (469h11m41.043536495s) [upload=411834]: Next returned nil segment Dec 25 04:40:41.172473798 (469h11m41.043556833s) [upload=411834]: Waiting for error group managing segments... Dec 25 04:40:41.483361556 (469h11m41.354444601s) [noise=true , address=145.79.227.37:4567 , piece_id=TZ4C57ISUI3CB3VOLTSM5W5QVDI7PVXV67PFJI3ZSVSXHTKPYPSA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.484283170 (469h11m41.355366205s) [noise=true , address=145.79.227.5:4221 , piece_id=C4SXUVTG2TZ4EIRBGL4TLOLBMYOAZ4KLIQSZOXKEOWAJBQQJLWSA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.515393699 (469h11m41.386476744s) [noise=true , address=145.79.227.51:4411 , piece_id=WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.526179387 (469h11m41.397262422s) [noise=true , address=45.91.116.5:4300 , piece_id=NTWC3RWYQQBBKPMXGUG44R2B5GWC2Z4LDFU3GTWOWCT3JIRIZOOQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.642380892 (469h11m41.513463937s) [noise=true , address=145.79.227.36:4517 , piece_id=HBQU7CMAO2KJS6ODW2WHICBAWQSH6TQQPOQSENKDEISWCWFN2CMA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.662326662 (469h11m41.533409707s) [noise=true , address=145.79.227.3:4150 , piece_id=OHGA4EA33AVEDCJLB6QSUJLD3W2TV6YNN6FBVFFKTTSF2B6XDPRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.758852130 (469h11m41.629935175s) [noise=true , address=145.79.227.47:4265 , piece_id=TVUXNR6K35A3CWFDN4JRKM4Q6RGBC4LP2XKPXJFN2AJL32ESEBPA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.823449082 (469h11m41.694532117s) [noise=true , address=145.79.227.17:4970 , piece_id=6DPKJ762IUN4A7UEZXLDERCVDW24QEKGEHFETAYZOKWQPVN2QJYA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.952636616 (469h11m41.823719661s) [noise=true , address=145.79.227.19:4079 , piece_id=ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.025177123 (469h11m41.896260168s) [noise=true , address=145.79.227.45:4229 , piece_id=GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.222526274 (469h11m42.093609319s) [noise=true , address=145.79.227.8:4331 , piece_id=ELLVNJFOVY5HSAHZDQYGQVO7OYHFZNFHCBJKHQFV5K6EV2M3JTRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.250024374 (469h11m42.121107409s) [noise=true , address=145.79.227.2:4114 , piece_id=5ISXZKQSH4CLLTF3ILGAXHV6STMLRUSZP6UATU2677X2FUPLBCRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.651787921 (469h11m42.522870966s) [noise=true , address=145.79.227.15:4875 , piece_id=FUMLT2ODHUE3ETSNWVKBYLSJXYVCHUW5KAJJE5AECNL3IAQYYCLQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.651814982 (469h11m42.522898027s) [seg_pos={0 0}, upload=411834]: Segment reached optimal threshold of 13 pieces. Dec 25 04:40:42.652052093 (469h11m42.523135129s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: write tcp 188.166.225.170:36650->77.90.31.15:4697: use of closed network connection; storage node overloaded, request limit: 2000 Dec 25 04:40:42.652069537 (469h11m42.523152572s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652177673 (469h11m42.523260708s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd): protocol: write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection; write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection Dec 25 04:40:42.652184456 (469h11m42.523267501s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652230243 (469h11m42.523313308s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq): protocol: write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection Dec 25 04:40:42.652240693 (469h11m42.523323778s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652268265 (469h11m42.52335131s) [seg_pos={0 0}, upload=411834]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 04:40:42.652354770 (469h11m42.523437815s) [upload=411834]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 04:40:42.652361693 (469h11m42.523444738s) [upload=411834]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 04:40:42.959704440 (469h11m42.830787485s) [upload=411834]: Done uploading segments... Dec 25 04:40:42.959728234 (469h11m42.83081127s) [upload=411834]: Upload finished. err: <nil> Duration: 5.878680478s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 5.980080985s trace: https://jaeger.dev.storj.io/trace/681cf842d2260d41 FAIL (2025-12-25 04:47:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, Piece ID: ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ) context deadline exceeded; piecestore: (Node ID: 1DT68m6J4PpDk3XUDhDaaqV46NLKVz4A23716JLFxEAh122GeR, Piece ID: WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA) context deadline exceeded; piecestore: (Node ID: 12wWYnhWHSqqAFgje4p9Cr7WYaJ6dJWsCqvrx8ZqVFoakjuFeka, Piece ID: GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q) context deadline exceeded ======LOGS====== nline: false Dec 25 04:40:39.039085560 (469h11m38.910168605s) [upload=411834]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 04:40:39.039095028 (469h11m38.910178073s) [upload=411834]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 04:40:39.297407613 (469h11m39.168490658s) [seg_pos={0 0}, upload=411834]: Begin upload segment... Dec 25 04:40:39.297471084 (469h11m39.168554119s) [seg_pos={0 0}, upload=411834]: Done begin upload segment. Dec 25 04:40:39.297511581 (469h11m39.168594616s) [noise=true , address=145.79.227.51:4411 , piece_id=WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.297663310 (469h11m39.168746345s) [noise=true , address=145.79.227.17:4970 , piece_id=6DPKJ762IUN4A7UEZXLDERCVDW24QEKGEHFETAYZOKWQPVN2QJYA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.298405574 (469h11m39.169488619s) [noise=true , address=145.79.227.2:4114 , piece_id=5ISXZKQSH4CLLTF3ILGAXHV6STMLRUSZP6UATU2677X2FUPLBCRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.298797961 (469h11m39.169881006s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299124392 (469h11m39.170207437s) [noise=true , address=45.91.116.5:4300 , piece_id=NTWC3RWYQQBBKPMXGUG44R2B5GWC2Z4LDFU3GTWOWCT3JIRIZOOQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299438650 (469h11m39.170521685s) [noise=true , address=145.79.227.5:4221 , piece_id=C4SXUVTG2TZ4EIRBGL4TLOLBMYOAZ4KLIQSZOXKEOWAJBQQJLWSA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299794709 (469h11m39.170877744s) [noise=true , address=145.79.227.37:4567 , piece_id=TZ4C57ISUI3CB3VOLTSM5W5QVDI7PVXV67PFJI3ZSVSXHTKPYPSA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.300142221 (469h11m39.171225256s) [noise=true , address=145.79.227.8:4331 , piece_id=ELLVNJFOVY5HSAHZDQYGQVO7OYHFZNFHCBJKHQFV5K6EV2M3JTRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.300506354 (469h11m39.171589389s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.301142795 (469h11m39.17222584s) [noise=true , address=145.79.227.36:4517 , piece_id=HBQU7CMAO2KJS6ODW2WHICBAWQSH6TQQPOQSENKDEISWCWFN2CMA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.301625835 (469h11m39.17270891s) [noise=true , address=145.79.227.45:4229 , piece_id=GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.302170964 (469h11m39.173254009s) [noise=true , address=145.79.227.3:4150 , piece_id=OHGA4EA33AVEDCJLB6QSUJLD3W2TV6YNN6FBVFFKTTSF2B6XDPRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.302629947 (469h11m39.173712982s) [noise=true , address=145.79.227.19:4079 , piece_id=ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303079704 (469h11m39.174162749s) [noise=true , address=145.79.227.15:4875 , piece_id=FUMLT2ODHUE3ETSNWVKBYLSJXYVCHUW5KAJJE5AECNL3IAQYYCLQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303499804 (469h11m39.174582849s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303921386 (469h11m39.175004432s) [noise=true , address=145.79.227.47:4265 , piece_id=TVUXNR6K35A3CWFDN4JRKM4Q6RGBC4LP2XKPXJFN2AJL32ESEBPA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:41.172453460 (469h11m41.043536495s) [upload=411834]: Next returned nil segment Dec 25 04:40:41.172473798 (469h11m41.043556833s) [upload=411834]: Waiting for error group managing segments... Dec 25 04:40:41.483361556 (469h11m41.354444601s) [noise=true , address=145.79.227.37:4567 , piece_id=TZ4C57ISUI3CB3VOLTSM5W5QVDI7PVXV67PFJI3ZSVSXHTKPYPSA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.484283170 (469h11m41.355366205s) [noise=true , address=145.79.227.5:4221 , piece_id=C4SXUVTG2TZ4EIRBGL4TLOLBMYOAZ4KLIQSZOXKEOWAJBQQJLWSA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.515393699 (469h11m41.386476744s) [noise=true , address=145.79.227.51:4411 , piece_id=WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.526179387 (469h11m41.397262422s) [noise=true , address=45.91.116.5:4300 , piece_id=NTWC3RWYQQBBKPMXGUG44R2B5GWC2Z4LDFU3GTWOWCT3JIRIZOOQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.642380892 (469h11m41.513463937s) [noise=true , address=145.79.227.36:4517 , piece_id=HBQU7CMAO2KJS6ODW2WHICBAWQSH6TQQPOQSENKDEISWCWFN2CMA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.662326662 (469h11m41.533409707s) [noise=true , address=145.79.227.3:4150 , piece_id=OHGA4EA33AVEDCJLB6QSUJLD3W2TV6YNN6FBVFFKTTSF2B6XDPRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.758852130 (469h11m41.629935175s) [noise=true , address=145.79.227.47:4265 , piece_id=TVUXNR6K35A3CWFDN4JRKM4Q6RGBC4LP2XKPXJFN2AJL32ESEBPA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.823449082 (469h11m41.694532117s) [noise=true , address=145.79.227.17:4970 , piece_id=6DPKJ762IUN4A7UEZXLDERCVDW24QEKGEHFETAYZOKWQPVN2QJYA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.952636616 (469h11m41.823719661s) [noise=true , address=145.79.227.19:4079 , piece_id=ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.025177123 (469h11m41.896260168s) [noise=true , address=145.79.227.45:4229 , piece_id=GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.222526274 (469h11m42.093609319s) [noise=true , address=145.79.227.8:4331 , piece_id=ELLVNJFOVY5HSAHZDQYGQVO7OYHFZNFHCBJKHQFV5K6EV2M3JTRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.250024374 (469h11m42.121107409s) [noise=true , address=145.79.227.2:4114 , piece_id=5ISXZKQSH4CLLTF3ILGAXHV6STMLRUSZP6UATU2677X2FUPLBCRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.651787921 (469h11m42.522870966s) [noise=true , address=145.79.227.15:4875 , piece_id=FUMLT2ODHUE3ETSNWVKBYLSJXYVCHUW5KAJJE5AECNL3IAQYYCLQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.651814982 (469h11m42.522898027s) [seg_pos={0 0}, upload=411834]: Segment reached optimal threshold of 13 pieces. Dec 25 04:40:42.652052093 (469h11m42.523135129s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: write tcp 188.166.225.170:36650->77.90.31.15:4697: use of closed network connection; storage node overloaded, request limit: 2000 Dec 25 04:40:42.652069537 (469h11m42.523152572s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652177673 (469h11m42.523260708s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd): protocol: write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection; write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection Dec 25 04:40:42.652184456 (469h11m42.523267501s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652230243 (469h11m42.523313308s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq): protocol: write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection Dec 25 04:40:42.652240693 (469h11m42.523323778s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652268265 (469h11m42.52335131s) [seg_pos={0 0}, upload=411834]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 04:40:42.652354770 (469h11m42.523437815s) [upload=411834]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 04:40:42.652361693 (469h11m42.523444738s) [upload=411834]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 04:40:42.959704440 (469h11m42.830787485s) [upload=411834]: Done uploading segments... Dec 25 04:40:42.959728234 (469h11m42.83081127s) [upload=411834]: Upload finished. err: <nil> Duration: 5.980108257s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 5.867222105s trace: https://jaeger.dev.storj.io/trace/681cf842d2260d41 FAIL (2025-12-25 04:46:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, Piece ID: ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ) context deadline exceeded; piecestore: (Node ID: 1DT68m6J4PpDk3XUDhDaaqV46NLKVz4A23716JLFxEAh122GeR, Piece ID: WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA) context deadline exceeded; piecestore: (Node ID: 12wWYnhWHSqqAFgje4p9Cr7WYaJ6dJWsCqvrx8ZqVFoakjuFeka, Piece ID: GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q) context deadline exceeded ======LOGS====== nline: false Dec 25 04:40:39.039085560 (469h11m38.910168605s) [upload=411834]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 04:40:39.039095028 (469h11m38.910178073s) [upload=411834]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 04:40:39.297407613 (469h11m39.168490658s) [seg_pos={0 0}, upload=411834]: Begin upload segment... Dec 25 04:40:39.297471084 (469h11m39.168554119s) [seg_pos={0 0}, upload=411834]: Done begin upload segment. Dec 25 04:40:39.297511581 (469h11m39.168594616s) [noise=true , address=145.79.227.51:4411 , piece_id=WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.297663310 (469h11m39.168746345s) [noise=true , address=145.79.227.17:4970 , piece_id=6DPKJ762IUN4A7UEZXLDERCVDW24QEKGEHFETAYZOKWQPVN2QJYA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.298405574 (469h11m39.169488619s) [noise=true , address=145.79.227.2:4114 , piece_id=5ISXZKQSH4CLLTF3ILGAXHV6STMLRUSZP6UATU2677X2FUPLBCRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.298797961 (469h11m39.169881006s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299124392 (469h11m39.170207437s) [noise=true , address=45.91.116.5:4300 , piece_id=NTWC3RWYQQBBKPMXGUG44R2B5GWC2Z4LDFU3GTWOWCT3JIRIZOOQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299438650 (469h11m39.170521685s) [noise=true , address=145.79.227.5:4221 , piece_id=C4SXUVTG2TZ4EIRBGL4TLOLBMYOAZ4KLIQSZOXKEOWAJBQQJLWSA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299794709 (469h11m39.170877744s) [noise=true , address=145.79.227.37:4567 , piece_id=TZ4C57ISUI3CB3VOLTSM5W5QVDI7PVXV67PFJI3ZSVSXHTKPYPSA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.300142221 (469h11m39.171225256s) [noise=true , address=145.79.227.8:4331 , piece_id=ELLVNJFOVY5HSAHZDQYGQVO7OYHFZNFHCBJKHQFV5K6EV2M3JTRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.300506354 (469h11m39.171589389s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.301142795 (469h11m39.17222584s) [noise=true , address=145.79.227.36:4517 , piece_id=HBQU7CMAO2KJS6ODW2WHICBAWQSH6TQQPOQSENKDEISWCWFN2CMA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.301625835 (469h11m39.17270891s) [noise=true , address=145.79.227.45:4229 , piece_id=GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.302170964 (469h11m39.173254009s) [noise=true , address=145.79.227.3:4150 , piece_id=OHGA4EA33AVEDCJLB6QSUJLD3W2TV6YNN6FBVFFKTTSF2B6XDPRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.302629947 (469h11m39.173712982s) [noise=true , address=145.79.227.19:4079 , piece_id=ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303079704 (469h11m39.174162749s) [noise=true , address=145.79.227.15:4875 , piece_id=FUMLT2ODHUE3ETSNWVKBYLSJXYVCHUW5KAJJE5AECNL3IAQYYCLQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303499804 (469h11m39.174582849s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303921386 (469h11m39.175004432s) [noise=true , address=145.79.227.47:4265 , piece_id=TVUXNR6K35A3CWFDN4JRKM4Q6RGBC4LP2XKPXJFN2AJL32ESEBPA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:41.172453460 (469h11m41.043536495s) [upload=411834]: Next returned nil segment Dec 25 04:40:41.172473798 (469h11m41.043556833s) [upload=411834]: Waiting for error group managing segments... Dec 25 04:40:41.483361556 (469h11m41.354444601s) [noise=true , address=145.79.227.37:4567 , piece_id=TZ4C57ISUI3CB3VOLTSM5W5QVDI7PVXV67PFJI3ZSVSXHTKPYPSA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.484283170 (469h11m41.355366205s) [noise=true , address=145.79.227.5:4221 , piece_id=C4SXUVTG2TZ4EIRBGL4TLOLBMYOAZ4KLIQSZOXKEOWAJBQQJLWSA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.515393699 (469h11m41.386476744s) [noise=true , address=145.79.227.51:4411 , piece_id=WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.526179387 (469h11m41.397262422s) [noise=true , address=45.91.116.5:4300 , piece_id=NTWC3RWYQQBBKPMXGUG44R2B5GWC2Z4LDFU3GTWOWCT3JIRIZOOQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.642380892 (469h11m41.513463937s) [noise=true , address=145.79.227.36:4517 , piece_id=HBQU7CMAO2KJS6ODW2WHICBAWQSH6TQQPOQSENKDEISWCWFN2CMA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.662326662 (469h11m41.533409707s) [noise=true , address=145.79.227.3:4150 , piece_id=OHGA4EA33AVEDCJLB6QSUJLD3W2TV6YNN6FBVFFKTTSF2B6XDPRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.758852130 (469h11m41.629935175s) [noise=true , address=145.79.227.47:4265 , piece_id=TVUXNR6K35A3CWFDN4JRKM4Q6RGBC4LP2XKPXJFN2AJL32ESEBPA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.823449082 (469h11m41.694532117s) [noise=true , address=145.79.227.17:4970 , piece_id=6DPKJ762IUN4A7UEZXLDERCVDW24QEKGEHFETAYZOKWQPVN2QJYA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.952636616 (469h11m41.823719661s) [noise=true , address=145.79.227.19:4079 , piece_id=ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.025177123 (469h11m41.896260168s) [noise=true , address=145.79.227.45:4229 , piece_id=GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.222526274 (469h11m42.093609319s) [noise=true , address=145.79.227.8:4331 , piece_id=ELLVNJFOVY5HSAHZDQYGQVO7OYHFZNFHCBJKHQFV5K6EV2M3JTRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.250024374 (469h11m42.121107409s) [noise=true , address=145.79.227.2:4114 , piece_id=5ISXZKQSH4CLLTF3ILGAXHV6STMLRUSZP6UATU2677X2FUPLBCRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.651787921 (469h11m42.522870966s) [noise=true , address=145.79.227.15:4875 , piece_id=FUMLT2ODHUE3ETSNWVKBYLSJXYVCHUW5KAJJE5AECNL3IAQYYCLQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.651814982 (469h11m42.522898027s) [seg_pos={0 0}, upload=411834]: Segment reached optimal threshold of 13 pieces. Dec 25 04:40:42.652052093 (469h11m42.523135129s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: write tcp 188.166.225.170:36650->77.90.31.15:4697: use of closed network connection; storage node overloaded, request limit: 2000 Dec 25 04:40:42.652069537 (469h11m42.523152572s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652177673 (469h11m42.523260708s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd): protocol: write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection; write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection Dec 25 04:40:42.652184456 (469h11m42.523267501s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652230243 (469h11m42.523313308s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq): protocol: write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection Dec 25 04:40:42.652240693 (469h11m42.523323778s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652268265 (469h11m42.52335131s) [seg_pos={0 0}, upload=411834]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 04:40:42.652354770 (469h11m42.523437815s) [upload=411834]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 04:40:42.652361693 (469h11m42.523444738s) [upload=411834]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 04:40:42.959704440 (469h11m42.830787485s) [upload=411834]: Done uploading segments... Dec 25 04:40:42.959728234 (469h11m42.83081127s) [upload=411834]: Upload finished. err: <nil> Duration: 5.867248696s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 5.900655128s trace: https://jaeger.dev.storj.io/trace/681cf842d2260d41 FAIL (2025-12-25 04:45:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, Piece ID: ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ) context deadline exceeded; piecestore: (Node ID: 1DT68m6J4PpDk3XUDhDaaqV46NLKVz4A23716JLFxEAh122GeR, Piece ID: WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA) context deadline exceeded; piecestore: (Node ID: 12wWYnhWHSqqAFgje4p9Cr7WYaJ6dJWsCqvrx8ZqVFoakjuFeka, Piece ID: GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q) context deadline exceeded ======LOGS====== nline: false Dec 25 04:40:39.039085560 (469h11m38.910168605s) [upload=411834]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 04:40:39.039095028 (469h11m38.910178073s) [upload=411834]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 04:40:39.297407613 (469h11m39.168490658s) [seg_pos={0 0}, upload=411834]: Begin upload segment... Dec 25 04:40:39.297471084 (469h11m39.168554119s) [seg_pos={0 0}, upload=411834]: Done begin upload segment. Dec 25 04:40:39.297511581 (469h11m39.168594616s) [noise=true , address=145.79.227.51:4411 , piece_id=WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.297663310 (469h11m39.168746345s) [noise=true , address=145.79.227.17:4970 , piece_id=6DPKJ762IUN4A7UEZXLDERCVDW24QEKGEHFETAYZOKWQPVN2QJYA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.298405574 (469h11m39.169488619s) [noise=true , address=145.79.227.2:4114 , piece_id=5ISXZKQSH4CLLTF3ILGAXHV6STMLRUSZP6UATU2677X2FUPLBCRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.298797961 (469h11m39.169881006s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299124392 (469h11m39.170207437s) [noise=true , address=45.91.116.5:4300 , piece_id=NTWC3RWYQQBBKPMXGUG44R2B5GWC2Z4LDFU3GTWOWCT3JIRIZOOQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299438650 (469h11m39.170521685s) [noise=true , address=145.79.227.5:4221 , piece_id=C4SXUVTG2TZ4EIRBGL4TLOLBMYOAZ4KLIQSZOXKEOWAJBQQJLWSA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299794709 (469h11m39.170877744s) [noise=true , address=145.79.227.37:4567 , piece_id=TZ4C57ISUI3CB3VOLTSM5W5QVDI7PVXV67PFJI3ZSVSXHTKPYPSA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.300142221 (469h11m39.171225256s) [noise=true , address=145.79.227.8:4331 , piece_id=ELLVNJFOVY5HSAHZDQYGQVO7OYHFZNFHCBJKHQFV5K6EV2M3JTRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.300506354 (469h11m39.171589389s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.301142795 (469h11m39.17222584s) [noise=true , address=145.79.227.36:4517 , piece_id=HBQU7CMAO2KJS6ODW2WHICBAWQSH6TQQPOQSENKDEISWCWFN2CMA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.301625835 (469h11m39.17270891s) [noise=true , address=145.79.227.45:4229 , piece_id=GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.302170964 (469h11m39.173254009s) [noise=true , address=145.79.227.3:4150 , piece_id=OHGA4EA33AVEDCJLB6QSUJLD3W2TV6YNN6FBVFFKTTSF2B6XDPRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.302629947 (469h11m39.173712982s) [noise=true , address=145.79.227.19:4079 , piece_id=ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303079704 (469h11m39.174162749s) [noise=true , address=145.79.227.15:4875 , piece_id=FUMLT2ODHUE3ETSNWVKBYLSJXYVCHUW5KAJJE5AECNL3IAQYYCLQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303499804 (469h11m39.174582849s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303921386 (469h11m39.175004432s) [noise=true , address=145.79.227.47:4265 , piece_id=TVUXNR6K35A3CWFDN4JRKM4Q6RGBC4LP2XKPXJFN2AJL32ESEBPA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:41.172453460 (469h11m41.043536495s) [upload=411834]: Next returned nil segment Dec 25 04:40:41.172473798 (469h11m41.043556833s) [upload=411834]: Waiting for error group managing segments... Dec 25 04:40:41.483361556 (469h11m41.354444601s) [noise=true , address=145.79.227.37:4567 , piece_id=TZ4C57ISUI3CB3VOLTSM5W5QVDI7PVXV67PFJI3ZSVSXHTKPYPSA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.484283170 (469h11m41.355366205s) [noise=true , address=145.79.227.5:4221 , piece_id=C4SXUVTG2TZ4EIRBGL4TLOLBMYOAZ4KLIQSZOXKEOWAJBQQJLWSA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.515393699 (469h11m41.386476744s) [noise=true , address=145.79.227.51:4411 , piece_id=WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.526179387 (469h11m41.397262422s) [noise=true , address=45.91.116.5:4300 , piece_id=NTWC3RWYQQBBKPMXGUG44R2B5GWC2Z4LDFU3GTWOWCT3JIRIZOOQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.642380892 (469h11m41.513463937s) [noise=true , address=145.79.227.36:4517 , piece_id=HBQU7CMAO2KJS6ODW2WHICBAWQSH6TQQPOQSENKDEISWCWFN2CMA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.662326662 (469h11m41.533409707s) [noise=true , address=145.79.227.3:4150 , piece_id=OHGA4EA33AVEDCJLB6QSUJLD3W2TV6YNN6FBVFFKTTSF2B6XDPRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.758852130 (469h11m41.629935175s) [noise=true , address=145.79.227.47:4265 , piece_id=TVUXNR6K35A3CWFDN4JRKM4Q6RGBC4LP2XKPXJFN2AJL32ESEBPA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.823449082 (469h11m41.694532117s) [noise=true , address=145.79.227.17:4970 , piece_id=6DPKJ762IUN4A7UEZXLDERCVDW24QEKGEHFETAYZOKWQPVN2QJYA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.952636616 (469h11m41.823719661s) [noise=true , address=145.79.227.19:4079 , piece_id=ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.025177123 (469h11m41.896260168s) [noise=true , address=145.79.227.45:4229 , piece_id=GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.222526274 (469h11m42.093609319s) [noise=true , address=145.79.227.8:4331 , piece_id=ELLVNJFOVY5HSAHZDQYGQVO7OYHFZNFHCBJKHQFV5K6EV2M3JTRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.250024374 (469h11m42.121107409s) [noise=true , address=145.79.227.2:4114 , piece_id=5ISXZKQSH4CLLTF3ILGAXHV6STMLRUSZP6UATU2677X2FUPLBCRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.651787921 (469h11m42.522870966s) [noise=true , address=145.79.227.15:4875 , piece_id=FUMLT2ODHUE3ETSNWVKBYLSJXYVCHUW5KAJJE5AECNL3IAQYYCLQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.651814982 (469h11m42.522898027s) [seg_pos={0 0}, upload=411834]: Segment reached optimal threshold of 13 pieces. Dec 25 04:40:42.652052093 (469h11m42.523135129s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: write tcp 188.166.225.170:36650->77.90.31.15:4697: use of closed network connection; storage node overloaded, request limit: 2000 Dec 25 04:40:42.652069537 (469h11m42.523152572s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652177673 (469h11m42.523260708s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd): protocol: write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection; write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection Dec 25 04:40:42.652184456 (469h11m42.523267501s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652230243 (469h11m42.523313308s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq): protocol: write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection Dec 25 04:40:42.652240693 (469h11m42.523323778s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652268265 (469h11m42.52335131s) [seg_pos={0 0}, upload=411834]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 04:40:42.652354770 (469h11m42.523437815s) [upload=411834]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 04:40:42.652361693 (469h11m42.523444738s) [upload=411834]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 04:40:42.959704440 (469h11m42.830787485s) [upload=411834]: Done uploading segments... Dec 25 04:40:42.959728234 (469h11m42.83081127s) [upload=411834]: Upload finished. err: <nil> Duration: 5.900683541s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [X...........................................................] duration: 3.628952325s trace: https://jaeger.dev.storj.io/trace/3e5857d495cc33e0 FAIL (2025-12-25 04:45:28 UTC) orbiter warning: context deadline exceeded Duration: 3.628976091s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 5.912523279s trace: https://jaeger.dev.storj.io/trace/681cf842d2260d41 FAIL (2025-12-25 04:44:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, Piece ID: ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ) context deadline exceeded; piecestore: (Node ID: 1DT68m6J4PpDk3XUDhDaaqV46NLKVz4A23716JLFxEAh122GeR, Piece ID: WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA) context deadline exceeded; piecestore: (Node ID: 12wWYnhWHSqqAFgje4p9Cr7WYaJ6dJWsCqvrx8ZqVFoakjuFeka, Piece ID: GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q) context deadline exceeded ======LOGS====== nline: false Dec 25 04:40:39.039085560 (469h11m38.910168605s) [upload=411834]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 04:40:39.039095028 (469h11m38.910178073s) [upload=411834]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 04:40:39.297407613 (469h11m39.168490658s) [seg_pos={0 0}, upload=411834]: Begin upload segment... Dec 25 04:40:39.297471084 (469h11m39.168554119s) [seg_pos={0 0}, upload=411834]: Done begin upload segment. Dec 25 04:40:39.297511581 (469h11m39.168594616s) [noise=true , address=145.79.227.51:4411 , piece_id=WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.297663310 (469h11m39.168746345s) [noise=true , address=145.79.227.17:4970 , piece_id=6DPKJ762IUN4A7UEZXLDERCVDW24QEKGEHFETAYZOKWQPVN2QJYA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.298405574 (469h11m39.169488619s) [noise=true , address=145.79.227.2:4114 , piece_id=5ISXZKQSH4CLLTF3ILGAXHV6STMLRUSZP6UATU2677X2FUPLBCRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.298797961 (469h11m39.169881006s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299124392 (469h11m39.170207437s) [noise=true , address=45.91.116.5:4300 , piece_id=NTWC3RWYQQBBKPMXGUG44R2B5GWC2Z4LDFU3GTWOWCT3JIRIZOOQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299438650 (469h11m39.170521685s) [noise=true , address=145.79.227.5:4221 , piece_id=C4SXUVTG2TZ4EIRBGL4TLOLBMYOAZ4KLIQSZOXKEOWAJBQQJLWSA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299794709 (469h11m39.170877744s) [noise=true , address=145.79.227.37:4567 , piece_id=TZ4C57ISUI3CB3VOLTSM5W5QVDI7PVXV67PFJI3ZSVSXHTKPYPSA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.300142221 (469h11m39.171225256s) [noise=true , address=145.79.227.8:4331 , piece_id=ELLVNJFOVY5HSAHZDQYGQVO7OYHFZNFHCBJKHQFV5K6EV2M3JTRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.300506354 (469h11m39.171589389s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.301142795 (469h11m39.17222584s) [noise=true , address=145.79.227.36:4517 , piece_id=HBQU7CMAO2KJS6ODW2WHICBAWQSH6TQQPOQSENKDEISWCWFN2CMA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.301625835 (469h11m39.17270891s) [noise=true , address=145.79.227.45:4229 , piece_id=GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.302170964 (469h11m39.173254009s) [noise=true , address=145.79.227.3:4150 , piece_id=OHGA4EA33AVEDCJLB6QSUJLD3W2TV6YNN6FBVFFKTTSF2B6XDPRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.302629947 (469h11m39.173712982s) [noise=true , address=145.79.227.19:4079 , piece_id=ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303079704 (469h11m39.174162749s) [noise=true , address=145.79.227.15:4875 , piece_id=FUMLT2ODHUE3ETSNWVKBYLSJXYVCHUW5KAJJE5AECNL3IAQYYCLQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303499804 (469h11m39.174582849s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303921386 (469h11m39.175004432s) [noise=true , address=145.79.227.47:4265 , piece_id=TVUXNR6K35A3CWFDN4JRKM4Q6RGBC4LP2XKPXJFN2AJL32ESEBPA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:41.172453460 (469h11m41.043536495s) [upload=411834]: Next returned nil segment Dec 25 04:40:41.172473798 (469h11m41.043556833s) [upload=411834]: Waiting for error group managing segments... Dec 25 04:40:41.483361556 (469h11m41.354444601s) [noise=true , address=145.79.227.37:4567 , piece_id=TZ4C57ISUI3CB3VOLTSM5W5QVDI7PVXV67PFJI3ZSVSXHTKPYPSA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.484283170 (469h11m41.355366205s) [noise=true , address=145.79.227.5:4221 , piece_id=C4SXUVTG2TZ4EIRBGL4TLOLBMYOAZ4KLIQSZOXKEOWAJBQQJLWSA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.515393699 (469h11m41.386476744s) [noise=true , address=145.79.227.51:4411 , piece_id=WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.526179387 (469h11m41.397262422s) [noise=true , address=45.91.116.5:4300 , piece_id=NTWC3RWYQQBBKPMXGUG44R2B5GWC2Z4LDFU3GTWOWCT3JIRIZOOQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.642380892 (469h11m41.513463937s) [noise=true , address=145.79.227.36:4517 , piece_id=HBQU7CMAO2KJS6ODW2WHICBAWQSH6TQQPOQSENKDEISWCWFN2CMA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.662326662 (469h11m41.533409707s) [noise=true , address=145.79.227.3:4150 , piece_id=OHGA4EA33AVEDCJLB6QSUJLD3W2TV6YNN6FBVFFKTTSF2B6XDPRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.758852130 (469h11m41.629935175s) [noise=true , address=145.79.227.47:4265 , piece_id=TVUXNR6K35A3CWFDN4JRKM4Q6RGBC4LP2XKPXJFN2AJL32ESEBPA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.823449082 (469h11m41.694532117s) [noise=true , address=145.79.227.17:4970 , piece_id=6DPKJ762IUN4A7UEZXLDERCVDW24QEKGEHFETAYZOKWQPVN2QJYA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.952636616 (469h11m41.823719661s) [noise=true , address=145.79.227.19:4079 , piece_id=ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.025177123 (469h11m41.896260168s) [noise=true , address=145.79.227.45:4229 , piece_id=GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.222526274 (469h11m42.093609319s) [noise=true , address=145.79.227.8:4331 , piece_id=ELLVNJFOVY5HSAHZDQYGQVO7OYHFZNFHCBJKHQFV5K6EV2M3JTRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.250024374 (469h11m42.121107409s) [noise=true , address=145.79.227.2:4114 , piece_id=5ISXZKQSH4CLLTF3ILGAXHV6STMLRUSZP6UATU2677X2FUPLBCRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.651787921 (469h11m42.522870966s) [noise=true , address=145.79.227.15:4875 , piece_id=FUMLT2ODHUE3ETSNWVKBYLSJXYVCHUW5KAJJE5AECNL3IAQYYCLQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.651814982 (469h11m42.522898027s) [seg_pos={0 0}, upload=411834]: Segment reached optimal threshold of 13 pieces. Dec 25 04:40:42.652052093 (469h11m42.523135129s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: write tcp 188.166.225.170:36650->77.90.31.15:4697: use of closed network connection; storage node overloaded, request limit: 2000 Dec 25 04:40:42.652069537 (469h11m42.523152572s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652177673 (469h11m42.523260708s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd): protocol: write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection; write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection Dec 25 04:40:42.652184456 (469h11m42.523267501s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652230243 (469h11m42.523313308s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq): protocol: write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection Dec 25 04:40:42.652240693 (469h11m42.523323778s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652268265 (469h11m42.52335131s) [seg_pos={0 0}, upload=411834]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 04:40:42.652354770 (469h11m42.523437815s) [upload=411834]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 04:40:42.652361693 (469h11m42.523444738s) [upload=411834]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 04:40:42.959704440 (469h11m42.830787485s) [upload=411834]: Done uploading segments... Dec 25 04:40:42.959728234 (469h11m42.83081127s) [upload=411834]: Upload finished. err: <nil> Duration: 5.912543779s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.X..........................................................] duration: 3.495007522s trace: https://jaeger.dev.storj.io/trace/3e5857d495cc33e0 FAIL (2025-12-25 04:44:50 UTC) orbiter warning: context deadline exceeded Duration: 3.495054681s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..X.........................................................] duration: 3.473035386s trace: https://jaeger.dev.storj.io/trace/3e5857d495cc33e0 FAIL (2025-12-25 04:44:27 UTC) orbiter warning: context deadline exceeded Duration: 3.4730626s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 5.664641705s trace: https://jaeger.dev.storj.io/trace/681cf842d2260d41 FAIL (2025-12-25 04:44:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, Piece ID: ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ) context deadline exceeded; piecestore: (Node ID: 1DT68m6J4PpDk3XUDhDaaqV46NLKVz4A23716JLFxEAh122GeR, Piece ID: WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA) context deadline exceeded; piecestore: (Node ID: 12wWYnhWHSqqAFgje4p9Cr7WYaJ6dJWsCqvrx8ZqVFoakjuFeka, Piece ID: GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q) context deadline exceeded ======LOGS====== nline: false Dec 25 04:40:39.039085560 (469h11m38.910168605s) [upload=411834]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 04:40:39.039095028 (469h11m38.910178073s) [upload=411834]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 04:40:39.297407613 (469h11m39.168490658s) [seg_pos={0 0}, upload=411834]: Begin upload segment... Dec 25 04:40:39.297471084 (469h11m39.168554119s) [seg_pos={0 0}, upload=411834]: Done begin upload segment. Dec 25 04:40:39.297511581 (469h11m39.168594616s) [noise=true , address=145.79.227.51:4411 , piece_id=WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.297663310 (469h11m39.168746345s) [noise=true , address=145.79.227.17:4970 , piece_id=6DPKJ762IUN4A7UEZXLDERCVDW24QEKGEHFETAYZOKWQPVN2QJYA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.298405574 (469h11m39.169488619s) [noise=true , address=145.79.227.2:4114 , piece_id=5ISXZKQSH4CLLTF3ILGAXHV6STMLRUSZP6UATU2677X2FUPLBCRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.298797961 (469h11m39.169881006s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299124392 (469h11m39.170207437s) [noise=true , address=45.91.116.5:4300 , piece_id=NTWC3RWYQQBBKPMXGUG44R2B5GWC2Z4LDFU3GTWOWCT3JIRIZOOQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299438650 (469h11m39.170521685s) [noise=true , address=145.79.227.5:4221 , piece_id=C4SXUVTG2TZ4EIRBGL4TLOLBMYOAZ4KLIQSZOXKEOWAJBQQJLWSA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299794709 (469h11m39.170877744s) [noise=true , address=145.79.227.37:4567 , piece_id=TZ4C57ISUI3CB3VOLTSM5W5QVDI7PVXV67PFJI3ZSVSXHTKPYPSA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.300142221 (469h11m39.171225256s) [noise=true , address=145.79.227.8:4331 , piece_id=ELLVNJFOVY5HSAHZDQYGQVO7OYHFZNFHCBJKHQFV5K6EV2M3JTRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.300506354 (469h11m39.171589389s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.301142795 (469h11m39.17222584s) [noise=true , address=145.79.227.36:4517 , piece_id=HBQU7CMAO2KJS6ODW2WHICBAWQSH6TQQPOQSENKDEISWCWFN2CMA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.301625835 (469h11m39.17270891s) [noise=true , address=145.79.227.45:4229 , piece_id=GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.302170964 (469h11m39.173254009s) [noise=true , address=145.79.227.3:4150 , piece_id=OHGA4EA33AVEDCJLB6QSUJLD3W2TV6YNN6FBVFFKTTSF2B6XDPRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.302629947 (469h11m39.173712982s) [noise=true , address=145.79.227.19:4079 , piece_id=ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303079704 (469h11m39.174162749s) [noise=true , address=145.79.227.15:4875 , piece_id=FUMLT2ODHUE3ETSNWVKBYLSJXYVCHUW5KAJJE5AECNL3IAQYYCLQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303499804 (469h11m39.174582849s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303921386 (469h11m39.175004432s) [noise=true , address=145.79.227.47:4265 , piece_id=TVUXNR6K35A3CWFDN4JRKM4Q6RGBC4LP2XKPXJFN2AJL32ESEBPA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:41.172453460 (469h11m41.043536495s) [upload=411834]: Next returned nil segment Dec 25 04:40:41.172473798 (469h11m41.043556833s) [upload=411834]: Waiting for error group managing segments... Dec 25 04:40:41.483361556 (469h11m41.354444601s) [noise=true , address=145.79.227.37:4567 , piece_id=TZ4C57ISUI3CB3VOLTSM5W5QVDI7PVXV67PFJI3ZSVSXHTKPYPSA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.484283170 (469h11m41.355366205s) [noise=true , address=145.79.227.5:4221 , piece_id=C4SXUVTG2TZ4EIRBGL4TLOLBMYOAZ4KLIQSZOXKEOWAJBQQJLWSA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.515393699 (469h11m41.386476744s) [noise=true , address=145.79.227.51:4411 , piece_id=WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.526179387 (469h11m41.397262422s) [noise=true , address=45.91.116.5:4300 , piece_id=NTWC3RWYQQBBKPMXGUG44R2B5GWC2Z4LDFU3GTWOWCT3JIRIZOOQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.642380892 (469h11m41.513463937s) [noise=true , address=145.79.227.36:4517 , piece_id=HBQU7CMAO2KJS6ODW2WHICBAWQSH6TQQPOQSENKDEISWCWFN2CMA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.662326662 (469h11m41.533409707s) [noise=true , address=145.79.227.3:4150 , piece_id=OHGA4EA33AVEDCJLB6QSUJLD3W2TV6YNN6FBVFFKTTSF2B6XDPRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.758852130 (469h11m41.629935175s) [noise=true , address=145.79.227.47:4265 , piece_id=TVUXNR6K35A3CWFDN4JRKM4Q6RGBC4LP2XKPXJFN2AJL32ESEBPA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.823449082 (469h11m41.694532117s) [noise=true , address=145.79.227.17:4970 , piece_id=6DPKJ762IUN4A7UEZXLDERCVDW24QEKGEHFETAYZOKWQPVN2QJYA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.952636616 (469h11m41.823719661s) [noise=true , address=145.79.227.19:4079 , piece_id=ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.025177123 (469h11m41.896260168s) [noise=true , address=145.79.227.45:4229 , piece_id=GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.222526274 (469h11m42.093609319s) [noise=true , address=145.79.227.8:4331 , piece_id=ELLVNJFOVY5HSAHZDQYGQVO7OYHFZNFHCBJKHQFV5K6EV2M3JTRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.250024374 (469h11m42.121107409s) [noise=true , address=145.79.227.2:4114 , piece_id=5ISXZKQSH4CLLTF3ILGAXHV6STMLRUSZP6UATU2677X2FUPLBCRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.651787921 (469h11m42.522870966s) [noise=true , address=145.79.227.15:4875 , piece_id=FUMLT2ODHUE3ETSNWVKBYLSJXYVCHUW5KAJJE5AECNL3IAQYYCLQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.651814982 (469h11m42.522898027s) [seg_pos={0 0}, upload=411834]: Segment reached optimal threshold of 13 pieces. Dec 25 04:40:42.652052093 (469h11m42.523135129s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: write tcp 188.166.225.170:36650->77.90.31.15:4697: use of closed network connection; storage node overloaded, request limit: 2000 Dec 25 04:40:42.652069537 (469h11m42.523152572s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652177673 (469h11m42.523260708s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd): protocol: write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection; write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection Dec 25 04:40:42.652184456 (469h11m42.523267501s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652230243 (469h11m42.523313308s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq): protocol: write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection Dec 25 04:40:42.652240693 (469h11m42.523323778s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652268265 (469h11m42.52335131s) [seg_pos={0 0}, upload=411834]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 04:40:42.652354770 (469h11m42.523437815s) [upload=411834]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 04:40:42.652361693 (469h11m42.523444738s) [upload=411834]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 04:40:42.959704440 (469h11m42.830787485s) [upload=411834]: Done uploading segments... Dec 25 04:40:42.959728234 (469h11m42.83081127s) [upload=411834]: Upload finished. err: <nil> Duration: 5.664667193s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 5.926534396s trace: https://jaeger.dev.storj.io/trace/681cf842d2260d41 FAIL (2025-12-25 04:43:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, Piece ID: ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ) context deadline exceeded; piecestore: (Node ID: 1DT68m6J4PpDk3XUDhDaaqV46NLKVz4A23716JLFxEAh122GeR, Piece ID: WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA) context deadline exceeded; piecestore: (Node ID: 12wWYnhWHSqqAFgje4p9Cr7WYaJ6dJWsCqvrx8ZqVFoakjuFeka, Piece ID: GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q) context deadline exceeded ======LOGS====== nline: false Dec 25 04:40:39.039085560 (469h11m38.910168605s) [upload=411834]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 04:40:39.039095028 (469h11m38.910178073s) [upload=411834]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 04:40:39.297407613 (469h11m39.168490658s) [seg_pos={0 0}, upload=411834]: Begin upload segment... Dec 25 04:40:39.297471084 (469h11m39.168554119s) [seg_pos={0 0}, upload=411834]: Done begin upload segment. Dec 25 04:40:39.297511581 (469h11m39.168594616s) [noise=true , address=145.79.227.51:4411 , piece_id=WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.297663310 (469h11m39.168746345s) [noise=true , address=145.79.227.17:4970 , piece_id=6DPKJ762IUN4A7UEZXLDERCVDW24QEKGEHFETAYZOKWQPVN2QJYA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.298405574 (469h11m39.169488619s) [noise=true , address=145.79.227.2:4114 , piece_id=5ISXZKQSH4CLLTF3ILGAXHV6STMLRUSZP6UATU2677X2FUPLBCRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.298797961 (469h11m39.169881006s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299124392 (469h11m39.170207437s) [noise=true , address=45.91.116.5:4300 , piece_id=NTWC3RWYQQBBKPMXGUG44R2B5GWC2Z4LDFU3GTWOWCT3JIRIZOOQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299438650 (469h11m39.170521685s) [noise=true , address=145.79.227.5:4221 , piece_id=C4SXUVTG2TZ4EIRBGL4TLOLBMYOAZ4KLIQSZOXKEOWAJBQQJLWSA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299794709 (469h11m39.170877744s) [noise=true , address=145.79.227.37:4567 , piece_id=TZ4C57ISUI3CB3VOLTSM5W5QVDI7PVXV67PFJI3ZSVSXHTKPYPSA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.300142221 (469h11m39.171225256s) [noise=true , address=145.79.227.8:4331 , piece_id=ELLVNJFOVY5HSAHZDQYGQVO7OYHFZNFHCBJKHQFV5K6EV2M3JTRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.300506354 (469h11m39.171589389s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.301142795 (469h11m39.17222584s) [noise=true , address=145.79.227.36:4517 , piece_id=HBQU7CMAO2KJS6ODW2WHICBAWQSH6TQQPOQSENKDEISWCWFN2CMA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.301625835 (469h11m39.17270891s) [noise=true , address=145.79.227.45:4229 , piece_id=GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.302170964 (469h11m39.173254009s) [noise=true , address=145.79.227.3:4150 , piece_id=OHGA4EA33AVEDCJLB6QSUJLD3W2TV6YNN6FBVFFKTTSF2B6XDPRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.302629947 (469h11m39.173712982s) [noise=true , address=145.79.227.19:4079 , piece_id=ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303079704 (469h11m39.174162749s) [noise=true , address=145.79.227.15:4875 , piece_id=FUMLT2ODHUE3ETSNWVKBYLSJXYVCHUW5KAJJE5AECNL3IAQYYCLQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303499804 (469h11m39.174582849s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303921386 (469h11m39.175004432s) [noise=true , address=145.79.227.47:4265 , piece_id=TVUXNR6K35A3CWFDN4JRKM4Q6RGBC4LP2XKPXJFN2AJL32ESEBPA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:41.172453460 (469h11m41.043536495s) [upload=411834]: Next returned nil segment Dec 25 04:40:41.172473798 (469h11m41.043556833s) [upload=411834]: Waiting for error group managing segments... Dec 25 04:40:41.483361556 (469h11m41.354444601s) [noise=true , address=145.79.227.37:4567 , piece_id=TZ4C57ISUI3CB3VOLTSM5W5QVDI7PVXV67PFJI3ZSVSXHTKPYPSA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.484283170 (469h11m41.355366205s) [noise=true , address=145.79.227.5:4221 , piece_id=C4SXUVTG2TZ4EIRBGL4TLOLBMYOAZ4KLIQSZOXKEOWAJBQQJLWSA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.515393699 (469h11m41.386476744s) [noise=true , address=145.79.227.51:4411 , piece_id=WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.526179387 (469h11m41.397262422s) [noise=true , address=45.91.116.5:4300 , piece_id=NTWC3RWYQQBBKPMXGUG44R2B5GWC2Z4LDFU3GTWOWCT3JIRIZOOQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.642380892 (469h11m41.513463937s) [noise=true , address=145.79.227.36:4517 , piece_id=HBQU7CMAO2KJS6ODW2WHICBAWQSH6TQQPOQSENKDEISWCWFN2CMA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.662326662 (469h11m41.533409707s) [noise=true , address=145.79.227.3:4150 , piece_id=OHGA4EA33AVEDCJLB6QSUJLD3W2TV6YNN6FBVFFKTTSF2B6XDPRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.758852130 (469h11m41.629935175s) [noise=true , address=145.79.227.47:4265 , piece_id=TVUXNR6K35A3CWFDN4JRKM4Q6RGBC4LP2XKPXJFN2AJL32ESEBPA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.823449082 (469h11m41.694532117s) [noise=true , address=145.79.227.17:4970 , piece_id=6DPKJ762IUN4A7UEZXLDERCVDW24QEKGEHFETAYZOKWQPVN2QJYA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.952636616 (469h11m41.823719661s) [noise=true , address=145.79.227.19:4079 , piece_id=ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.025177123 (469h11m41.896260168s) [noise=true , address=145.79.227.45:4229 , piece_id=GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.222526274 (469h11m42.093609319s) [noise=true , address=145.79.227.8:4331 , piece_id=ELLVNJFOVY5HSAHZDQYGQVO7OYHFZNFHCBJKHQFV5K6EV2M3JTRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.250024374 (469h11m42.121107409s) [noise=true , address=145.79.227.2:4114 , piece_id=5ISXZKQSH4CLLTF3ILGAXHV6STMLRUSZP6UATU2677X2FUPLBCRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.651787921 (469h11m42.522870966s) [noise=true , address=145.79.227.15:4875 , piece_id=FUMLT2ODHUE3ETSNWVKBYLSJXYVCHUW5KAJJE5AECNL3IAQYYCLQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.651814982 (469h11m42.522898027s) [seg_pos={0 0}, upload=411834]: Segment reached optimal threshold of 13 pieces. Dec 25 04:40:42.652052093 (469h11m42.523135129s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: write tcp 188.166.225.170:36650->77.90.31.15:4697: use of closed network connection; storage node overloaded, request limit: 2000 Dec 25 04:40:42.652069537 (469h11m42.523152572s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652177673 (469h11m42.523260708s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd): protocol: write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection; write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection Dec 25 04:40:42.652184456 (469h11m42.523267501s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652230243 (469h11m42.523313308s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq): protocol: write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection Dec 25 04:40:42.652240693 (469h11m42.523323778s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652268265 (469h11m42.52335131s) [seg_pos={0 0}, upload=411834]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 04:40:42.652354770 (469h11m42.523437815s) [upload=411834]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 04:40:42.652361693 (469h11m42.523444738s) [upload=411834]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 04:40:42.959704440 (469h11m42.830787485s) [upload=411834]: Done uploading segments... Dec 25 04:40:42.959728234 (469h11m42.83081127s) [upload=411834]: Upload finished. err: <nil> Duration: 5.926569082s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...X........................................................] duration: 3.428151431s trace: https://jaeger.dev.storj.io/trace/3e5857d495cc33e0 FAIL (2025-12-25 04:43:40 UTC) orbiter warning: context deadline exceeded Duration: 3.428181358s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 5.856100119s trace: https://jaeger.dev.storj.io/trace/681cf842d2260d41 FAIL (2025-12-25 04:43:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, Piece ID: ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ) context deadline exceeded; piecestore: (Node ID: 1DT68m6J4PpDk3XUDhDaaqV46NLKVz4A23716JLFxEAh122GeR, Piece ID: WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA) context deadline exceeded; piecestore: (Node ID: 12wWYnhWHSqqAFgje4p9Cr7WYaJ6dJWsCqvrx8ZqVFoakjuFeka, Piece ID: GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q) context deadline exceeded ======LOGS====== nline: false Dec 25 04:40:39.039085560 (469h11m38.910168605s) [upload=411834]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 04:40:39.039095028 (469h11m38.910178073s) [upload=411834]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 04:40:39.297407613 (469h11m39.168490658s) [seg_pos={0 0}, upload=411834]: Begin upload segment... Dec 25 04:40:39.297471084 (469h11m39.168554119s) [seg_pos={0 0}, upload=411834]: Done begin upload segment. Dec 25 04:40:39.297511581 (469h11m39.168594616s) [noise=true , address=145.79.227.51:4411 , piece_id=WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.297663310 (469h11m39.168746345s) [noise=true , address=145.79.227.17:4970 , piece_id=6DPKJ762IUN4A7UEZXLDERCVDW24QEKGEHFETAYZOKWQPVN2QJYA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.298405574 (469h11m39.169488619s) [noise=true , address=145.79.227.2:4114 , piece_id=5ISXZKQSH4CLLTF3ILGAXHV6STMLRUSZP6UATU2677X2FUPLBCRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.298797961 (469h11m39.169881006s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299124392 (469h11m39.170207437s) [noise=true , address=45.91.116.5:4300 , piece_id=NTWC3RWYQQBBKPMXGUG44R2B5GWC2Z4LDFU3GTWOWCT3JIRIZOOQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299438650 (469h11m39.170521685s) [noise=true , address=145.79.227.5:4221 , piece_id=C4SXUVTG2TZ4EIRBGL4TLOLBMYOAZ4KLIQSZOXKEOWAJBQQJLWSA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299794709 (469h11m39.170877744s) [noise=true , address=145.79.227.37:4567 , piece_id=TZ4C57ISUI3CB3VOLTSM5W5QVDI7PVXV67PFJI3ZSVSXHTKPYPSA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.300142221 (469h11m39.171225256s) [noise=true , address=145.79.227.8:4331 , piece_id=ELLVNJFOVY5HSAHZDQYGQVO7OYHFZNFHCBJKHQFV5K6EV2M3JTRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.300506354 (469h11m39.171589389s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.301142795 (469h11m39.17222584s) [noise=true , address=145.79.227.36:4517 , piece_id=HBQU7CMAO2KJS6ODW2WHICBAWQSH6TQQPOQSENKDEISWCWFN2CMA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.301625835 (469h11m39.17270891s) [noise=true , address=145.79.227.45:4229 , piece_id=GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.302170964 (469h11m39.173254009s) [noise=true , address=145.79.227.3:4150 , piece_id=OHGA4EA33AVEDCJLB6QSUJLD3W2TV6YNN6FBVFFKTTSF2B6XDPRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.302629947 (469h11m39.173712982s) [noise=true , address=145.79.227.19:4079 , piece_id=ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303079704 (469h11m39.174162749s) [noise=true , address=145.79.227.15:4875 , piece_id=FUMLT2ODHUE3ETSNWVKBYLSJXYVCHUW5KAJJE5AECNL3IAQYYCLQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303499804 (469h11m39.174582849s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303921386 (469h11m39.175004432s) [noise=true , address=145.79.227.47:4265 , piece_id=TVUXNR6K35A3CWFDN4JRKM4Q6RGBC4LP2XKPXJFN2AJL32ESEBPA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:41.172453460 (469h11m41.043536495s) [upload=411834]: Next returned nil segment Dec 25 04:40:41.172473798 (469h11m41.043556833s) [upload=411834]: Waiting for error group managing segments... Dec 25 04:40:41.483361556 (469h11m41.354444601s) [noise=true , address=145.79.227.37:4567 , piece_id=TZ4C57ISUI3CB3VOLTSM5W5QVDI7PVXV67PFJI3ZSVSXHTKPYPSA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.484283170 (469h11m41.355366205s) [noise=true , address=145.79.227.5:4221 , piece_id=C4SXUVTG2TZ4EIRBGL4TLOLBMYOAZ4KLIQSZOXKEOWAJBQQJLWSA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.515393699 (469h11m41.386476744s) [noise=true , address=145.79.227.51:4411 , piece_id=WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.526179387 (469h11m41.397262422s) [noise=true , address=45.91.116.5:4300 , piece_id=NTWC3RWYQQBBKPMXGUG44R2B5GWC2Z4LDFU3GTWOWCT3JIRIZOOQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.642380892 (469h11m41.513463937s) [noise=true , address=145.79.227.36:4517 , piece_id=HBQU7CMAO2KJS6ODW2WHICBAWQSH6TQQPOQSENKDEISWCWFN2CMA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.662326662 (469h11m41.533409707s) [noise=true , address=145.79.227.3:4150 , piece_id=OHGA4EA33AVEDCJLB6QSUJLD3W2TV6YNN6FBVFFKTTSF2B6XDPRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.758852130 (469h11m41.629935175s) [noise=true , address=145.79.227.47:4265 , piece_id=TVUXNR6K35A3CWFDN4JRKM4Q6RGBC4LP2XKPXJFN2AJL32ESEBPA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.823449082 (469h11m41.694532117s) [noise=true , address=145.79.227.17:4970 , piece_id=6DPKJ762IUN4A7UEZXLDERCVDW24QEKGEHFETAYZOKWQPVN2QJYA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.952636616 (469h11m41.823719661s) [noise=true , address=145.79.227.19:4079 , piece_id=ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.025177123 (469h11m41.896260168s) [noise=true , address=145.79.227.45:4229 , piece_id=GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.222526274 (469h11m42.093609319s) [noise=true , address=145.79.227.8:4331 , piece_id=ELLVNJFOVY5HSAHZDQYGQVO7OYHFZNFHCBJKHQFV5K6EV2M3JTRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.250024374 (469h11m42.121107409s) [noise=true , address=145.79.227.2:4114 , piece_id=5ISXZKQSH4CLLTF3ILGAXHV6STMLRUSZP6UATU2677X2FUPLBCRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.651787921 (469h11m42.522870966s) [noise=true , address=145.79.227.15:4875 , piece_id=FUMLT2ODHUE3ETSNWVKBYLSJXYVCHUW5KAJJE5AECNL3IAQYYCLQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.651814982 (469h11m42.522898027s) [seg_pos={0 0}, upload=411834]: Segment reached optimal threshold of 13 pieces. Dec 25 04:40:42.652052093 (469h11m42.523135129s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: write tcp 188.166.225.170:36650->77.90.31.15:4697: use of closed network connection; storage node overloaded, request limit: 2000 Dec 25 04:40:42.652069537 (469h11m42.523152572s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652177673 (469h11m42.523260708s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd): protocol: write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection; write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection Dec 25 04:40:42.652184456 (469h11m42.523267501s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652230243 (469h11m42.523313308s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq): protocol: write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection Dec 25 04:40:42.652240693 (469h11m42.523323778s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652268265 (469h11m42.52335131s) [seg_pos={0 0}, upload=411834]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 04:40:42.652354770 (469h11m42.523437815s) [upload=411834]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 04:40:42.652361693 (469h11m42.523444738s) [upload=411834]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 04:40:42.959704440 (469h11m42.830787485s) [upload=411834]: Done uploading segments... Dec 25 04:40:42.959728234 (469h11m42.83081127s) [upload=411834]: Upload finished. err: <nil> Duration: 5.856131396s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 5.907276938s trace: https://jaeger.dev.storj.io/trace/681cf842d2260d41 FAIL (2025-12-25 04:42:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, Piece ID: ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ) context deadline exceeded; piecestore: (Node ID: 1DT68m6J4PpDk3XUDhDaaqV46NLKVz4A23716JLFxEAh122GeR, Piece ID: WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA) context deadline exceeded; piecestore: (Node ID: 12wWYnhWHSqqAFgje4p9Cr7WYaJ6dJWsCqvrx8ZqVFoakjuFeka, Piece ID: GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q) context deadline exceeded ======LOGS====== nline: false Dec 25 04:40:39.039085560 (469h11m38.910168605s) [upload=411834]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 04:40:39.039095028 (469h11m38.910178073s) [upload=411834]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 04:40:39.297407613 (469h11m39.168490658s) [seg_pos={0 0}, upload=411834]: Begin upload segment... Dec 25 04:40:39.297471084 (469h11m39.168554119s) [seg_pos={0 0}, upload=411834]: Done begin upload segment. Dec 25 04:40:39.297511581 (469h11m39.168594616s) [noise=true , address=145.79.227.51:4411 , piece_id=WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.297663310 (469h11m39.168746345s) [noise=true , address=145.79.227.17:4970 , piece_id=6DPKJ762IUN4A7UEZXLDERCVDW24QEKGEHFETAYZOKWQPVN2QJYA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.298405574 (469h11m39.169488619s) [noise=true , address=145.79.227.2:4114 , piece_id=5ISXZKQSH4CLLTF3ILGAXHV6STMLRUSZP6UATU2677X2FUPLBCRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.298797961 (469h11m39.169881006s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299124392 (469h11m39.170207437s) [noise=true , address=45.91.116.5:4300 , piece_id=NTWC3RWYQQBBKPMXGUG44R2B5GWC2Z4LDFU3GTWOWCT3JIRIZOOQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299438650 (469h11m39.170521685s) [noise=true , address=145.79.227.5:4221 , piece_id=C4SXUVTG2TZ4EIRBGL4TLOLBMYOAZ4KLIQSZOXKEOWAJBQQJLWSA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299794709 (469h11m39.170877744s) [noise=true , address=145.79.227.37:4567 , piece_id=TZ4C57ISUI3CB3VOLTSM5W5QVDI7PVXV67PFJI3ZSVSXHTKPYPSA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.300142221 (469h11m39.171225256s) [noise=true , address=145.79.227.8:4331 , piece_id=ELLVNJFOVY5HSAHZDQYGQVO7OYHFZNFHCBJKHQFV5K6EV2M3JTRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.300506354 (469h11m39.171589389s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.301142795 (469h11m39.17222584s) [noise=true , address=145.79.227.36:4517 , piece_id=HBQU7CMAO2KJS6ODW2WHICBAWQSH6TQQPOQSENKDEISWCWFN2CMA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.301625835 (469h11m39.17270891s) [noise=true , address=145.79.227.45:4229 , piece_id=GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.302170964 (469h11m39.173254009s) [noise=true , address=145.79.227.3:4150 , piece_id=OHGA4EA33AVEDCJLB6QSUJLD3W2TV6YNN6FBVFFKTTSF2B6XDPRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.302629947 (469h11m39.173712982s) [noise=true , address=145.79.227.19:4079 , piece_id=ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303079704 (469h11m39.174162749s) [noise=true , address=145.79.227.15:4875 , piece_id=FUMLT2ODHUE3ETSNWVKBYLSJXYVCHUW5KAJJE5AECNL3IAQYYCLQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303499804 (469h11m39.174582849s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303921386 (469h11m39.175004432s) [noise=true , address=145.79.227.47:4265 , piece_id=TVUXNR6K35A3CWFDN4JRKM4Q6RGBC4LP2XKPXJFN2AJL32ESEBPA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:41.172453460 (469h11m41.043536495s) [upload=411834]: Next returned nil segment Dec 25 04:40:41.172473798 (469h11m41.043556833s) [upload=411834]: Waiting for error group managing segments... Dec 25 04:40:41.483361556 (469h11m41.354444601s) [noise=true , address=145.79.227.37:4567 , piece_id=TZ4C57ISUI3CB3VOLTSM5W5QVDI7PVXV67PFJI3ZSVSXHTKPYPSA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.484283170 (469h11m41.355366205s) [noise=true , address=145.79.227.5:4221 , piece_id=C4SXUVTG2TZ4EIRBGL4TLOLBMYOAZ4KLIQSZOXKEOWAJBQQJLWSA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.515393699 (469h11m41.386476744s) [noise=true , address=145.79.227.51:4411 , piece_id=WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.526179387 (469h11m41.397262422s) [noise=true , address=45.91.116.5:4300 , piece_id=NTWC3RWYQQBBKPMXGUG44R2B5GWC2Z4LDFU3GTWOWCT3JIRIZOOQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.642380892 (469h11m41.513463937s) [noise=true , address=145.79.227.36:4517 , piece_id=HBQU7CMAO2KJS6ODW2WHICBAWQSH6TQQPOQSENKDEISWCWFN2CMA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.662326662 (469h11m41.533409707s) [noise=true , address=145.79.227.3:4150 , piece_id=OHGA4EA33AVEDCJLB6QSUJLD3W2TV6YNN6FBVFFKTTSF2B6XDPRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.758852130 (469h11m41.629935175s) [noise=true , address=145.79.227.47:4265 , piece_id=TVUXNR6K35A3CWFDN4JRKM4Q6RGBC4LP2XKPXJFN2AJL32ESEBPA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.823449082 (469h11m41.694532117s) [noise=true , address=145.79.227.17:4970 , piece_id=6DPKJ762IUN4A7UEZXLDERCVDW24QEKGEHFETAYZOKWQPVN2QJYA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.952636616 (469h11m41.823719661s) [noise=true , address=145.79.227.19:4079 , piece_id=ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.025177123 (469h11m41.896260168s) [noise=true , address=145.79.227.45:4229 , piece_id=GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.222526274 (469h11m42.093609319s) [noise=true , address=145.79.227.8:4331 , piece_id=ELLVNJFOVY5HSAHZDQYGQVO7OYHFZNFHCBJKHQFV5K6EV2M3JTRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.250024374 (469h11m42.121107409s) [noise=true , address=145.79.227.2:4114 , piece_id=5ISXZKQSH4CLLTF3ILGAXHV6STMLRUSZP6UATU2677X2FUPLBCRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.651787921 (469h11m42.522870966s) [noise=true , address=145.79.227.15:4875 , piece_id=FUMLT2ODHUE3ETSNWVKBYLSJXYVCHUW5KAJJE5AECNL3IAQYYCLQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.651814982 (469h11m42.522898027s) [seg_pos={0 0}, upload=411834]: Segment reached optimal threshold of 13 pieces. Dec 25 04:40:42.652052093 (469h11m42.523135129s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: write tcp 188.166.225.170:36650->77.90.31.15:4697: use of closed network connection; storage node overloaded, request limit: 2000 Dec 25 04:40:42.652069537 (469h11m42.523152572s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652177673 (469h11m42.523260708s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd): protocol: write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection; write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection Dec 25 04:40:42.652184456 (469h11m42.523267501s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652230243 (469h11m42.523313308s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq): protocol: write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection Dec 25 04:40:42.652240693 (469h11m42.523323778s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652268265 (469h11m42.52335131s) [seg_pos={0 0}, upload=411834]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 04:40:42.652354770 (469h11m42.523437815s) [upload=411834]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 04:40:42.652361693 (469h11m42.523444738s) [upload=411834]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 04:40:42.959704440 (469h11m42.830787485s) [upload=411834]: Done uploading segments... Dec 25 04:40:42.959728234 (469h11m42.83081127s) [upload=411834]: Upload finished. err: <nil> Duration: 5.907305361s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....X.......................................................] duration: 3.469476081s trace: https://jaeger.dev.storj.io/trace/3e5857d495cc33e0 FAIL (2025-12-25 04:42:35 UTC) orbiter warning: context deadline exceeded Duration: 3.469550232s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 7.993130233s trace: https://jaeger.dev.storj.io/trace/681cf842d2260d41 FAIL (2025-12-25 04:42:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, Piece ID: ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ) context deadline exceeded; piecestore: (Node ID: 1DT68m6J4PpDk3XUDhDaaqV46NLKVz4A23716JLFxEAh122GeR, Piece ID: WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA) context deadline exceeded; piecestore: (Node ID: 12wWYnhWHSqqAFgje4p9Cr7WYaJ6dJWsCqvrx8ZqVFoakjuFeka, Piece ID: GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q) context deadline exceeded ======LOGS====== nline: false Dec 25 04:40:39.039085560 (469h11m38.910168605s) [upload=411834]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 04:40:39.039095028 (469h11m38.910178073s) [upload=411834]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 04:40:39.297407613 (469h11m39.168490658s) [seg_pos={0 0}, upload=411834]: Begin upload segment... Dec 25 04:40:39.297471084 (469h11m39.168554119s) [seg_pos={0 0}, upload=411834]: Done begin upload segment. Dec 25 04:40:39.297511581 (469h11m39.168594616s) [noise=true , address=145.79.227.51:4411 , piece_id=WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.297663310 (469h11m39.168746345s) [noise=true , address=145.79.227.17:4970 , piece_id=6DPKJ762IUN4A7UEZXLDERCVDW24QEKGEHFETAYZOKWQPVN2QJYA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.298405574 (469h11m39.169488619s) [noise=true , address=145.79.227.2:4114 , piece_id=5ISXZKQSH4CLLTF3ILGAXHV6STMLRUSZP6UATU2677X2FUPLBCRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.298797961 (469h11m39.169881006s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299124392 (469h11m39.170207437s) [noise=true , address=45.91.116.5:4300 , piece_id=NTWC3RWYQQBBKPMXGUG44R2B5GWC2Z4LDFU3GTWOWCT3JIRIZOOQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299438650 (469h11m39.170521685s) [noise=true , address=145.79.227.5:4221 , piece_id=C4SXUVTG2TZ4EIRBGL4TLOLBMYOAZ4KLIQSZOXKEOWAJBQQJLWSA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299794709 (469h11m39.170877744s) [noise=true , address=145.79.227.37:4567 , piece_id=TZ4C57ISUI3CB3VOLTSM5W5QVDI7PVXV67PFJI3ZSVSXHTKPYPSA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.300142221 (469h11m39.171225256s) [noise=true , address=145.79.227.8:4331 , piece_id=ELLVNJFOVY5HSAHZDQYGQVO7OYHFZNFHCBJKHQFV5K6EV2M3JTRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.300506354 (469h11m39.171589389s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.301142795 (469h11m39.17222584s) [noise=true , address=145.79.227.36:4517 , piece_id=HBQU7CMAO2KJS6ODW2WHICBAWQSH6TQQPOQSENKDEISWCWFN2CMA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.301625835 (469h11m39.17270891s) [noise=true , address=145.79.227.45:4229 , piece_id=GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.302170964 (469h11m39.173254009s) [noise=true , address=145.79.227.3:4150 , piece_id=OHGA4EA33AVEDCJLB6QSUJLD3W2TV6YNN6FBVFFKTTSF2B6XDPRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.302629947 (469h11m39.173712982s) [noise=true , address=145.79.227.19:4079 , piece_id=ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303079704 (469h11m39.174162749s) [noise=true , address=145.79.227.15:4875 , piece_id=FUMLT2ODHUE3ETSNWVKBYLSJXYVCHUW5KAJJE5AECNL3IAQYYCLQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303499804 (469h11m39.174582849s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303921386 (469h11m39.175004432s) [noise=true , address=145.79.227.47:4265 , piece_id=TVUXNR6K35A3CWFDN4JRKM4Q6RGBC4LP2XKPXJFN2AJL32ESEBPA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:41.172453460 (469h11m41.043536495s) [upload=411834]: Next returned nil segment Dec 25 04:40:41.172473798 (469h11m41.043556833s) [upload=411834]: Waiting for error group managing segments... Dec 25 04:40:41.483361556 (469h11m41.354444601s) [noise=true , address=145.79.227.37:4567 , piece_id=TZ4C57ISUI3CB3VOLTSM5W5QVDI7PVXV67PFJI3ZSVSXHTKPYPSA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.484283170 (469h11m41.355366205s) [noise=true , address=145.79.227.5:4221 , piece_id=C4SXUVTG2TZ4EIRBGL4TLOLBMYOAZ4KLIQSZOXKEOWAJBQQJLWSA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.515393699 (469h11m41.386476744s) [noise=true , address=145.79.227.51:4411 , piece_id=WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.526179387 (469h11m41.397262422s) [noise=true , address=45.91.116.5:4300 , piece_id=NTWC3RWYQQBBKPMXGUG44R2B5GWC2Z4LDFU3GTWOWCT3JIRIZOOQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.642380892 (469h11m41.513463937s) [noise=true , address=145.79.227.36:4517 , piece_id=HBQU7CMAO2KJS6ODW2WHICBAWQSH6TQQPOQSENKDEISWCWFN2CMA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.662326662 (469h11m41.533409707s) [noise=true , address=145.79.227.3:4150 , piece_id=OHGA4EA33AVEDCJLB6QSUJLD3W2TV6YNN6FBVFFKTTSF2B6XDPRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.758852130 (469h11m41.629935175s) [noise=true , address=145.79.227.47:4265 , piece_id=TVUXNR6K35A3CWFDN4JRKM4Q6RGBC4LP2XKPXJFN2AJL32ESEBPA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.823449082 (469h11m41.694532117s) [noise=true , address=145.79.227.17:4970 , piece_id=6DPKJ762IUN4A7UEZXLDERCVDW24QEKGEHFETAYZOKWQPVN2QJYA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.952636616 (469h11m41.823719661s) [noise=true , address=145.79.227.19:4079 , piece_id=ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.025177123 (469h11m41.896260168s) [noise=true , address=145.79.227.45:4229 , piece_id=GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.222526274 (469h11m42.093609319s) [noise=true , address=145.79.227.8:4331 , piece_id=ELLVNJFOVY5HSAHZDQYGQVO7OYHFZNFHCBJKHQFV5K6EV2M3JTRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.250024374 (469h11m42.121107409s) [noise=true , address=145.79.227.2:4114 , piece_id=5ISXZKQSH4CLLTF3ILGAXHV6STMLRUSZP6UATU2677X2FUPLBCRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.651787921 (469h11m42.522870966s) [noise=true , address=145.79.227.15:4875 , piece_id=FUMLT2ODHUE3ETSNWVKBYLSJXYVCHUW5KAJJE5AECNL3IAQYYCLQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.651814982 (469h11m42.522898027s) [seg_pos={0 0}, upload=411834]: Segment reached optimal threshold of 13 pieces. Dec 25 04:40:42.652052093 (469h11m42.523135129s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: write tcp 188.166.225.170:36650->77.90.31.15:4697: use of closed network connection; storage node overloaded, request limit: 2000 Dec 25 04:40:42.652069537 (469h11m42.523152572s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652177673 (469h11m42.523260708s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd): protocol: write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection; write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection Dec 25 04:40:42.652184456 (469h11m42.523267501s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652230243 (469h11m42.523313308s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq): protocol: write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection Dec 25 04:40:42.652240693 (469h11m42.523323778s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652268265 (469h11m42.52335131s) [seg_pos={0 0}, upload=411834]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 04:40:42.652354770 (469h11m42.523437815s) [upload=411834]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 04:40:42.652361693 (469h11m42.523444738s) [upload=411834]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 04:40:42.959704440 (469h11m42.830787485s) [upload=411834]: Done uploading segments... Dec 25 04:40:42.959728234 (469h11m42.83081127s) [upload=411834]: Upload finished. err: <nil> Duration: 7.993158857s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....X......................................................] duration: 3.566086902s trace: https://jaeger.dev.storj.io/trace/3e5857d495cc33e0 FAIL (2025-12-25 04:41:54 UTC) orbiter warning: context deadline exceeded Duration: 3.566110006s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 20.746086275s trace: https://jaeger.dev.storj.io/trace/681cf842d2260d41 FAIL (2025-12-25 04:41:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, Piece ID: ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ) context deadline exceeded; piecestore: (Node ID: 1DT68m6J4PpDk3XUDhDaaqV46NLKVz4A23716JLFxEAh122GeR, Piece ID: WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA) context deadline exceeded; piecestore: (Node ID: 12wWYnhWHSqqAFgje4p9Cr7WYaJ6dJWsCqvrx8ZqVFoakjuFeka, Piece ID: GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q) context deadline exceeded ======LOGS====== nline: false Dec 25 04:40:39.039085560 (469h11m38.910168605s) [upload=411834]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 04:40:39.039095028 (469h11m38.910178073s) [upload=411834]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 04:40:39.297407613 (469h11m39.168490658s) [seg_pos={0 0}, upload=411834]: Begin upload segment... Dec 25 04:40:39.297471084 (469h11m39.168554119s) [seg_pos={0 0}, upload=411834]: Done begin upload segment. Dec 25 04:40:39.297511581 (469h11m39.168594616s) [noise=true , address=145.79.227.51:4411 , piece_id=WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.297663310 (469h11m39.168746345s) [noise=true , address=145.79.227.17:4970 , piece_id=6DPKJ762IUN4A7UEZXLDERCVDW24QEKGEHFETAYZOKWQPVN2QJYA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.298405574 (469h11m39.169488619s) [noise=true , address=145.79.227.2:4114 , piece_id=5ISXZKQSH4CLLTF3ILGAXHV6STMLRUSZP6UATU2677X2FUPLBCRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.298797961 (469h11m39.169881006s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299124392 (469h11m39.170207437s) [noise=true , address=45.91.116.5:4300 , piece_id=NTWC3RWYQQBBKPMXGUG44R2B5GWC2Z4LDFU3GTWOWCT3JIRIZOOQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299438650 (469h11m39.170521685s) [noise=true , address=145.79.227.5:4221 , piece_id=C4SXUVTG2TZ4EIRBGL4TLOLBMYOAZ4KLIQSZOXKEOWAJBQQJLWSA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299794709 (469h11m39.170877744s) [noise=true , address=145.79.227.37:4567 , piece_id=TZ4C57ISUI3CB3VOLTSM5W5QVDI7PVXV67PFJI3ZSVSXHTKPYPSA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.300142221 (469h11m39.171225256s) [noise=true , address=145.79.227.8:4331 , piece_id=ELLVNJFOVY5HSAHZDQYGQVO7OYHFZNFHCBJKHQFV5K6EV2M3JTRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.300506354 (469h11m39.171589389s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.301142795 (469h11m39.17222584s) [noise=true , address=145.79.227.36:4517 , piece_id=HBQU7CMAO2KJS6ODW2WHICBAWQSH6TQQPOQSENKDEISWCWFN2CMA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.301625835 (469h11m39.17270891s) [noise=true , address=145.79.227.45:4229 , piece_id=GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.302170964 (469h11m39.173254009s) [noise=true , address=145.79.227.3:4150 , piece_id=OHGA4EA33AVEDCJLB6QSUJLD3W2TV6YNN6FBVFFKTTSF2B6XDPRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.302629947 (469h11m39.173712982s) [noise=true , address=145.79.227.19:4079 , piece_id=ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303079704 (469h11m39.174162749s) [noise=true , address=145.79.227.15:4875 , piece_id=FUMLT2ODHUE3ETSNWVKBYLSJXYVCHUW5KAJJE5AECNL3IAQYYCLQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303499804 (469h11m39.174582849s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303921386 (469h11m39.175004432s) [noise=true , address=145.79.227.47:4265 , piece_id=TVUXNR6K35A3CWFDN4JRKM4Q6RGBC4LP2XKPXJFN2AJL32ESEBPA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:41.172453460 (469h11m41.043536495s) [upload=411834]: Next returned nil segment Dec 25 04:40:41.172473798 (469h11m41.043556833s) [upload=411834]: Waiting for error group managing segments... Dec 25 04:40:41.483361556 (469h11m41.354444601s) [noise=true , address=145.79.227.37:4567 , piece_id=TZ4C57ISUI3CB3VOLTSM5W5QVDI7PVXV67PFJI3ZSVSXHTKPYPSA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.484283170 (469h11m41.355366205s) [noise=true , address=145.79.227.5:4221 , piece_id=C4SXUVTG2TZ4EIRBGL4TLOLBMYOAZ4KLIQSZOXKEOWAJBQQJLWSA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.515393699 (469h11m41.386476744s) [noise=true , address=145.79.227.51:4411 , piece_id=WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.526179387 (469h11m41.397262422s) [noise=true , address=45.91.116.5:4300 , piece_id=NTWC3RWYQQBBKPMXGUG44R2B5GWC2Z4LDFU3GTWOWCT3JIRIZOOQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.642380892 (469h11m41.513463937s) [noise=true , address=145.79.227.36:4517 , piece_id=HBQU7CMAO2KJS6ODW2WHICBAWQSH6TQQPOQSENKDEISWCWFN2CMA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.662326662 (469h11m41.533409707s) [noise=true , address=145.79.227.3:4150 , piece_id=OHGA4EA33AVEDCJLB6QSUJLD3W2TV6YNN6FBVFFKTTSF2B6XDPRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.758852130 (469h11m41.629935175s) [noise=true , address=145.79.227.47:4265 , piece_id=TVUXNR6K35A3CWFDN4JRKM4Q6RGBC4LP2XKPXJFN2AJL32ESEBPA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.823449082 (469h11m41.694532117s) [noise=true , address=145.79.227.17:4970 , piece_id=6DPKJ762IUN4A7UEZXLDERCVDW24QEKGEHFETAYZOKWQPVN2QJYA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.952636616 (469h11m41.823719661s) [noise=true , address=145.79.227.19:4079 , piece_id=ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.025177123 (469h11m41.896260168s) [noise=true , address=145.79.227.45:4229 , piece_id=GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.222526274 (469h11m42.093609319s) [noise=true , address=145.79.227.8:4331 , piece_id=ELLVNJFOVY5HSAHZDQYGQVO7OYHFZNFHCBJKHQFV5K6EV2M3JTRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.250024374 (469h11m42.121107409s) [noise=true , address=145.79.227.2:4114 , piece_id=5ISXZKQSH4CLLTF3ILGAXHV6STMLRUSZP6UATU2677X2FUPLBCRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.651787921 (469h11m42.522870966s) [noise=true , address=145.79.227.15:4875 , piece_id=FUMLT2ODHUE3ETSNWVKBYLSJXYVCHUW5KAJJE5AECNL3IAQYYCLQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.651814982 (469h11m42.522898027s) [seg_pos={0 0}, upload=411834]: Segment reached optimal threshold of 13 pieces. Dec 25 04:40:42.652052093 (469h11m42.523135129s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: write tcp 188.166.225.170:36650->77.90.31.15:4697: use of closed network connection; storage node overloaded, request limit: 2000 Dec 25 04:40:42.652069537 (469h11m42.523152572s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652177673 (469h11m42.523260708s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd): protocol: write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection; write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection Dec 25 04:40:42.652184456 (469h11m42.523267501s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652230243 (469h11m42.523313308s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq): protocol: write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection Dec 25 04:40:42.652240693 (469h11m42.523323778s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652268265 (469h11m42.52335131s) [seg_pos={0 0}, upload=411834]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 04:40:42.652354770 (469h11m42.523437815s) [upload=411834]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 04:40:42.652361693 (469h11m42.523444738s) [upload=411834]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 04:40:42.959704440 (469h11m42.830787485s) [upload=411834]: Done uploading segments... Dec 25 04:40:42.959728234 (469h11m42.83081127s) [upload=411834]: Upload finished. err: <nil> Duration: 20.746112545s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.00251934s trace: https://jaeger.dev.storj.io/trace/681cf842d2260d41 FAIL (2025-12-25 04:41:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, Piece ID: ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ) context deadline exceeded; piecestore: (Node ID: 1DT68m6J4PpDk3XUDhDaaqV46NLKVz4A23716JLFxEAh122GeR, Piece ID: WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA) context deadline exceeded; piecestore: (Node ID: 12wWYnhWHSqqAFgje4p9Cr7WYaJ6dJWsCqvrx8ZqVFoakjuFeka, Piece ID: GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q) context deadline exceeded ======LOGS====== nline: false Dec 25 04:40:39.039085560 (469h11m38.910168605s) [upload=411834]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 04:40:39.039095028 (469h11m38.910178073s) [upload=411834]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 04:40:39.297407613 (469h11m39.168490658s) [seg_pos={0 0}, upload=411834]: Begin upload segment... Dec 25 04:40:39.297471084 (469h11m39.168554119s) [seg_pos={0 0}, upload=411834]: Done begin upload segment. Dec 25 04:40:39.297511581 (469h11m39.168594616s) [noise=true , address=145.79.227.51:4411 , piece_id=WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.297663310 (469h11m39.168746345s) [noise=true , address=145.79.227.17:4970 , piece_id=6DPKJ762IUN4A7UEZXLDERCVDW24QEKGEHFETAYZOKWQPVN2QJYA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.298405574 (469h11m39.169488619s) [noise=true , address=145.79.227.2:4114 , piece_id=5ISXZKQSH4CLLTF3ILGAXHV6STMLRUSZP6UATU2677X2FUPLBCRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.298797961 (469h11m39.169881006s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299124392 (469h11m39.170207437s) [noise=true , address=45.91.116.5:4300 , piece_id=NTWC3RWYQQBBKPMXGUG44R2B5GWC2Z4LDFU3GTWOWCT3JIRIZOOQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299438650 (469h11m39.170521685s) [noise=true , address=145.79.227.5:4221 , piece_id=C4SXUVTG2TZ4EIRBGL4TLOLBMYOAZ4KLIQSZOXKEOWAJBQQJLWSA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299794709 (469h11m39.170877744s) [noise=true , address=145.79.227.37:4567 , piece_id=TZ4C57ISUI3CB3VOLTSM5W5QVDI7PVXV67PFJI3ZSVSXHTKPYPSA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.300142221 (469h11m39.171225256s) [noise=true , address=145.79.227.8:4331 , piece_id=ELLVNJFOVY5HSAHZDQYGQVO7OYHFZNFHCBJKHQFV5K6EV2M3JTRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.300506354 (469h11m39.171589389s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.301142795 (469h11m39.17222584s) [noise=true , address=145.79.227.36:4517 , piece_id=HBQU7CMAO2KJS6ODW2WHICBAWQSH6TQQPOQSENKDEISWCWFN2CMA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.301625835 (469h11m39.17270891s) [noise=true , address=145.79.227.45:4229 , piece_id=GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.302170964 (469h11m39.173254009s) [noise=true , address=145.79.227.3:4150 , piece_id=OHGA4EA33AVEDCJLB6QSUJLD3W2TV6YNN6FBVFFKTTSF2B6XDPRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.302629947 (469h11m39.173712982s) [noise=true , address=145.79.227.19:4079 , piece_id=ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303079704 (469h11m39.174162749s) [noise=true , address=145.79.227.15:4875 , piece_id=FUMLT2ODHUE3ETSNWVKBYLSJXYVCHUW5KAJJE5AECNL3IAQYYCLQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303499804 (469h11m39.174582849s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303921386 (469h11m39.175004432s) [noise=true , address=145.79.227.47:4265 , piece_id=TVUXNR6K35A3CWFDN4JRKM4Q6RGBC4LP2XKPXJFN2AJL32ESEBPA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:41.172453460 (469h11m41.043536495s) [upload=411834]: Next returned nil segment Dec 25 04:40:41.172473798 (469h11m41.043556833s) [upload=411834]: Waiting for error group managing segments... Dec 25 04:40:41.483361556 (469h11m41.354444601s) [noise=true , address=145.79.227.37:4567 , piece_id=TZ4C57ISUI3CB3VOLTSM5W5QVDI7PVXV67PFJI3ZSVSXHTKPYPSA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.484283170 (469h11m41.355366205s) [noise=true , address=145.79.227.5:4221 , piece_id=C4SXUVTG2TZ4EIRBGL4TLOLBMYOAZ4KLIQSZOXKEOWAJBQQJLWSA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.515393699 (469h11m41.386476744s) [noise=true , address=145.79.227.51:4411 , piece_id=WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.526179387 (469h11m41.397262422s) [noise=true , address=45.91.116.5:4300 , piece_id=NTWC3RWYQQBBKPMXGUG44R2B5GWC2Z4LDFU3GTWOWCT3JIRIZOOQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.642380892 (469h11m41.513463937s) [noise=true , address=145.79.227.36:4517 , piece_id=HBQU7CMAO2KJS6ODW2WHICBAWQSH6TQQPOQSENKDEISWCWFN2CMA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.662326662 (469h11m41.533409707s) [noise=true , address=145.79.227.3:4150 , piece_id=OHGA4EA33AVEDCJLB6QSUJLD3W2TV6YNN6FBVFFKTTSF2B6XDPRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.758852130 (469h11m41.629935175s) [noise=true , address=145.79.227.47:4265 , piece_id=TVUXNR6K35A3CWFDN4JRKM4Q6RGBC4LP2XKPXJFN2AJL32ESEBPA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.823449082 (469h11m41.694532117s) [noise=true , address=145.79.227.17:4970 , piece_id=6DPKJ762IUN4A7UEZXLDERCVDW24QEKGEHFETAYZOKWQPVN2QJYA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.952636616 (469h11m41.823719661s) [noise=true , address=145.79.227.19:4079 , piece_id=ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.025177123 (469h11m41.896260168s) [noise=true , address=145.79.227.45:4229 , piece_id=GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.222526274 (469h11m42.093609319s) [noise=true , address=145.79.227.8:4331 , piece_id=ELLVNJFOVY5HSAHZDQYGQVO7OYHFZNFHCBJKHQFV5K6EV2M3JTRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.250024374 (469h11m42.121107409s) [noise=true , address=145.79.227.2:4114 , piece_id=5ISXZKQSH4CLLTF3ILGAXHV6STMLRUSZP6UATU2677X2FUPLBCRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.651787921 (469h11m42.522870966s) [noise=true , address=145.79.227.15:4875 , piece_id=FUMLT2ODHUE3ETSNWVKBYLSJXYVCHUW5KAJJE5AECNL3IAQYYCLQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.651814982 (469h11m42.522898027s) [seg_pos={0 0}, upload=411834]: Segment reached optimal threshold of 13 pieces. Dec 25 04:40:42.652052093 (469h11m42.523135129s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: write tcp 188.166.225.170:36650->77.90.31.15:4697: use of closed network connection; storage node overloaded, request limit: 2000 Dec 25 04:40:42.652069537 (469h11m42.523152572s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652177673 (469h11m42.523260708s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd): protocol: write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection; write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection Dec 25 04:40:42.652184456 (469h11m42.523267501s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652230243 (469h11m42.523313308s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq): protocol: write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection Dec 25 04:40:42.652240693 (469h11m42.523323778s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652268265 (469h11m42.52335131s) [seg_pos={0 0}, upload=411834]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 04:40:42.652354770 (469h11m42.523437815s) [upload=411834]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 04:40:42.652361693 (469h11m42.523444738s) [upload=411834]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 04:40:42.959704440 (469h11m42.830787485s) [upload=411834]: Done uploading segments... Dec 25 04:40:42.959728234 (469h11m42.83081127s) [upload=411834]: Upload finished. err: <nil> Duration: 30.002536514s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......X.....................................................] duration: 3.786069269s trace: https://jaeger.dev.storj.io/trace/3e5857d495cc33e0 FAIL (2025-12-25 04:41:03 UTC) orbiter warning: context deadline exceeded Duration: 3.786098714s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......X....................................................] duration: 3.589448942s trace: https://jaeger.dev.storj.io/trace/3e5857d495cc33e0 FAIL (2025-12-25 04:40:11 UTC) orbiter warning: context deadline exceeded Duration: 3.589955627s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........X...................................................] duration: 3.62932705s trace: https://jaeger.dev.storj.io/trace/3e5857d495cc33e0 FAIL (2025-12-25 04:39:09 UTC) orbiter warning: context deadline exceeded Duration: 3.629366013s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........X..................................................] duration: 3.560713517s trace: https://jaeger.dev.storj.io/trace/3e5857d495cc33e0 FAIL (2025-12-25 04:38:36 UTC) orbiter warning: context deadline exceeded Duration: 3.560753093s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........X.................................................] duration: 3.566612738s trace: https://jaeger.dev.storj.io/trace/3e5857d495cc33e0 FAIL (2025-12-25 04:37:42 UTC) orbiter warning: context deadline exceeded Duration: 3.566635181s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........X................................................] duration: 3.6486148s trace: https://jaeger.dev.storj.io/trace/3e5857d495cc33e0 FAIL (2025-12-25 04:36:45 UTC) orbiter warning: context deadline exceeded Duration: 3.648643425s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [............X...............................................] duration: 3.597318852s trace: https://jaeger.dev.storj.io/trace/3e5857d495cc33e0 FAIL (2025-12-25 04:35:44 UTC) orbiter warning: context deadline exceeded Duration: 3.597348327s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.............X..............................................] duration: 3.53635423s trace: https://jaeger.dev.storj.io/trace/3e5857d495cc33e0 FAIL (2025-12-25 04:34:37 UTC) orbiter warning: context deadline exceeded Duration: 3.536378496s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..............X.............................................] duration: 3.381583178s trace: https://jaeger.dev.storj.io/trace/3e5857d495cc33e0 FAIL (2025-12-25 04:33:32 UTC) orbiter warning: context deadline exceeded Duration: 3.381606733s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...............X............................................] duration: 3.590382562s trace: https://jaeger.dev.storj.io/trace/3e5857d495cc33e0 FAIL (2025-12-25 04:32:44 UTC) orbiter warning: context deadline exceeded Duration: 3.590408331s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [................X...........................................] duration: 3.604539708s trace: https://jaeger.dev.storj.io/trace/3e5857d495cc33e0 FAIL (2025-12-25 04:31:45 UTC) orbiter warning: context deadline exceeded Duration: 3.604567611s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.................X..........................................] duration: 3.56040539s trace: https://jaeger.dev.storj.io/trace/3e5857d495cc33e0 FAIL (2025-12-25 04:31:09 UTC) orbiter warning: context deadline exceeded Duration: 3.560476654s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)