Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........................................X.................] duration: 3.537711419s trace: https://jaeger.dev.storj.io/trace/835f8cfb18d0547 FAIL (2025-12-25 05:28:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.537740965s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........................................X................] duration: 3.436042141s trace: https://jaeger.dev.storj.io/trace/835f8cfb18d0547 FAIL (2025-12-25 05:28:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.436066407s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [............................................X...............] duration: 3.505298866s trace: https://jaeger.dev.storj.io/trace/835f8cfb18d0547 FAIL (2025-12-25 05:27:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.505319324s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.............................................X..............] duration: 3.543655087s trace: https://jaeger.dev.storj.io/trace/835f8cfb18d0547 FAIL (2025-12-25 05:26:40 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.543680257s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..............................................X.............] duration: 3.497995567s trace: https://jaeger.dev.storj.io/trace/835f8cfb18d0547 FAIL (2025-12-25 05:26:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.49802359s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...............................................X............] duration: 3.640037459s trace: https://jaeger.dev.storj.io/trace/835f8cfb18d0547 FAIL (2025-12-25 05:25:14 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.640068348s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [................................................X...........] duration: 3.475338572s trace: https://jaeger.dev.storj.io/trace/835f8cfb18d0547 FAIL (2025-12-25 05:24:33 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.475372517s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.................................................X..........] duration: 3.503880424s trace: https://jaeger.dev.storj.io/trace/835f8cfb18d0547 FAIL (2025-12-25 05:23:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.503905443s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 15.031076377s trace: https://jaeger.dev.storj.io/trace/681cf842d2260d41 FAIL (2025-12-25 05:23:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, Piece ID: ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ) context deadline exceeded; piecestore: (Node ID: 1DT68m6J4PpDk3XUDhDaaqV46NLKVz4A23716JLFxEAh122GeR, Piece ID: WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA) context deadline exceeded; piecestore: (Node ID: 12wWYnhWHSqqAFgje4p9Cr7WYaJ6dJWsCqvrx8ZqVFoakjuFeka, Piece ID: GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q) context deadline exceeded ======LOGS====== nline: false Dec 25 04:40:39.039085560 (469h11m38.910168605s) [upload=411834]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 04:40:39.039095028 (469h11m38.910178073s) [upload=411834]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 04:40:39.297407613 (469h11m39.168490658s) [seg_pos={0 0}, upload=411834]: Begin upload segment... Dec 25 04:40:39.297471084 (469h11m39.168554119s) [seg_pos={0 0}, upload=411834]: Done begin upload segment. Dec 25 04:40:39.297511581 (469h11m39.168594616s) [noise=true , address=145.79.227.51:4411 , piece_id=WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.297663310 (469h11m39.168746345s) [noise=true , address=145.79.227.17:4970 , piece_id=6DPKJ762IUN4A7UEZXLDERCVDW24QEKGEHFETAYZOKWQPVN2QJYA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.298405574 (469h11m39.169488619s) [noise=true , address=145.79.227.2:4114 , piece_id=5ISXZKQSH4CLLTF3ILGAXHV6STMLRUSZP6UATU2677X2FUPLBCRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.298797961 (469h11m39.169881006s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299124392 (469h11m39.170207437s) [noise=true , address=45.91.116.5:4300 , piece_id=NTWC3RWYQQBBKPMXGUG44R2B5GWC2Z4LDFU3GTWOWCT3JIRIZOOQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299438650 (469h11m39.170521685s) [noise=true , address=145.79.227.5:4221 , piece_id=C4SXUVTG2TZ4EIRBGL4TLOLBMYOAZ4KLIQSZOXKEOWAJBQQJLWSA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299794709 (469h11m39.170877744s) [noise=true , address=145.79.227.37:4567 , piece_id=TZ4C57ISUI3CB3VOLTSM5W5QVDI7PVXV67PFJI3ZSVSXHTKPYPSA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.300142221 (469h11m39.171225256s) [noise=true , address=145.79.227.8:4331 , piece_id=ELLVNJFOVY5HSAHZDQYGQVO7OYHFZNFHCBJKHQFV5K6EV2M3JTRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.300506354 (469h11m39.171589389s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.301142795 (469h11m39.17222584s) [noise=true , address=145.79.227.36:4517 , piece_id=HBQU7CMAO2KJS6ODW2WHICBAWQSH6TQQPOQSENKDEISWCWFN2CMA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.301625835 (469h11m39.17270891s) [noise=true , address=145.79.227.45:4229 , piece_id=GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.302170964 (469h11m39.173254009s) [noise=true , address=145.79.227.3:4150 , piece_id=OHGA4EA33AVEDCJLB6QSUJLD3W2TV6YNN6FBVFFKTTSF2B6XDPRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.302629947 (469h11m39.173712982s) [noise=true , address=145.79.227.19:4079 , piece_id=ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303079704 (469h11m39.174162749s) [noise=true , address=145.79.227.15:4875 , piece_id=FUMLT2ODHUE3ETSNWVKBYLSJXYVCHUW5KAJJE5AECNL3IAQYYCLQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303499804 (469h11m39.174582849s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303921386 (469h11m39.175004432s) [noise=true , address=145.79.227.47:4265 , piece_id=TVUXNR6K35A3CWFDN4JRKM4Q6RGBC4LP2XKPXJFN2AJL32ESEBPA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:41.172453460 (469h11m41.043536495s) [upload=411834]: Next returned nil segment Dec 25 04:40:41.172473798 (469h11m41.043556833s) [upload=411834]: Waiting for error group managing segments... Dec 25 04:40:41.483361556 (469h11m41.354444601s) [noise=true , address=145.79.227.37:4567 , piece_id=TZ4C57ISUI3CB3VOLTSM5W5QVDI7PVXV67PFJI3ZSVSXHTKPYPSA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.484283170 (469h11m41.355366205s) [noise=true , address=145.79.227.5:4221 , piece_id=C4SXUVTG2TZ4EIRBGL4TLOLBMYOAZ4KLIQSZOXKEOWAJBQQJLWSA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.515393699 (469h11m41.386476744s) [noise=true , address=145.79.227.51:4411 , piece_id=WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.526179387 (469h11m41.397262422s) [noise=true , address=45.91.116.5:4300 , piece_id=NTWC3RWYQQBBKPMXGUG44R2B5GWC2Z4LDFU3GTWOWCT3JIRIZOOQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.642380892 (469h11m41.513463937s) [noise=true , address=145.79.227.36:4517 , piece_id=HBQU7CMAO2KJS6ODW2WHICBAWQSH6TQQPOQSENKDEISWCWFN2CMA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.662326662 (469h11m41.533409707s) [noise=true , address=145.79.227.3:4150 , piece_id=OHGA4EA33AVEDCJLB6QSUJLD3W2TV6YNN6FBVFFKTTSF2B6XDPRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.758852130 (469h11m41.629935175s) [noise=true , address=145.79.227.47:4265 , piece_id=TVUXNR6K35A3CWFDN4JRKM4Q6RGBC4LP2XKPXJFN2AJL32ESEBPA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.823449082 (469h11m41.694532117s) [noise=true , address=145.79.227.17:4970 , piece_id=6DPKJ762IUN4A7UEZXLDERCVDW24QEKGEHFETAYZOKWQPVN2QJYA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.952636616 (469h11m41.823719661s) [noise=true , address=145.79.227.19:4079 , piece_id=ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.025177123 (469h11m41.896260168s) [noise=true , address=145.79.227.45:4229 , piece_id=GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.222526274 (469h11m42.093609319s) [noise=true , address=145.79.227.8:4331 , piece_id=ELLVNJFOVY5HSAHZDQYGQVO7OYHFZNFHCBJKHQFV5K6EV2M3JTRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.250024374 (469h11m42.121107409s) [noise=true , address=145.79.227.2:4114 , piece_id=5ISXZKQSH4CLLTF3ILGAXHV6STMLRUSZP6UATU2677X2FUPLBCRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.651787921 (469h11m42.522870966s) [noise=true , address=145.79.227.15:4875 , piece_id=FUMLT2ODHUE3ETSNWVKBYLSJXYVCHUW5KAJJE5AECNL3IAQYYCLQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.651814982 (469h11m42.522898027s) [seg_pos={0 0}, upload=411834]: Segment reached optimal threshold of 13 pieces. Dec 25 04:40:42.652052093 (469h11m42.523135129s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: write tcp 188.166.225.170:36650->77.90.31.15:4697: use of closed network connection; storage node overloaded, request limit: 2000 Dec 25 04:40:42.652069537 (469h11m42.523152572s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652177673 (469h11m42.523260708s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd): protocol: write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection; write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection Dec 25 04:40:42.652184456 (469h11m42.523267501s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652230243 (469h11m42.523313308s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq): protocol: write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection Dec 25 04:40:42.652240693 (469h11m42.523323778s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652268265 (469h11m42.52335131s) [seg_pos={0 0}, upload=411834]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 04:40:42.652354770 (469h11m42.523437815s) [upload=411834]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 04:40:42.652361693 (469h11m42.523444738s) [upload=411834]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 04:40:42.959704440 (469h11m42.830787485s) [upload=411834]: Done uploading segments... Dec 25 04:40:42.959728234 (469h11m42.83081127s) [upload=411834]: Upload finished. err: <nil> Duration: 15.031103729s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 5.68475226s trace: https://jaeger.dev.storj.io/trace/681cf842d2260d41 FAIL (2025-12-25 05:22:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, Piece ID: ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ) context deadline exceeded; piecestore: (Node ID: 1DT68m6J4PpDk3XUDhDaaqV46NLKVz4A23716JLFxEAh122GeR, Piece ID: WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA) context deadline exceeded; piecestore: (Node ID: 12wWYnhWHSqqAFgje4p9Cr7WYaJ6dJWsCqvrx8ZqVFoakjuFeka, Piece ID: GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q) context deadline exceeded ======LOGS====== nline: false Dec 25 04:40:39.039085560 (469h11m38.910168605s) [upload=411834]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 04:40:39.039095028 (469h11m38.910178073s) [upload=411834]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 04:40:39.297407613 (469h11m39.168490658s) [seg_pos={0 0}, upload=411834]: Begin upload segment... Dec 25 04:40:39.297471084 (469h11m39.168554119s) [seg_pos={0 0}, upload=411834]: Done begin upload segment. Dec 25 04:40:39.297511581 (469h11m39.168594616s) [noise=true , address=145.79.227.51:4411 , piece_id=WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.297663310 (469h11m39.168746345s) [noise=true , address=145.79.227.17:4970 , piece_id=6DPKJ762IUN4A7UEZXLDERCVDW24QEKGEHFETAYZOKWQPVN2QJYA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.298405574 (469h11m39.169488619s) [noise=true , address=145.79.227.2:4114 , piece_id=5ISXZKQSH4CLLTF3ILGAXHV6STMLRUSZP6UATU2677X2FUPLBCRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.298797961 (469h11m39.169881006s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299124392 (469h11m39.170207437s) [noise=true , address=45.91.116.5:4300 , piece_id=NTWC3RWYQQBBKPMXGUG44R2B5GWC2Z4LDFU3GTWOWCT3JIRIZOOQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299438650 (469h11m39.170521685s) [noise=true , address=145.79.227.5:4221 , piece_id=C4SXUVTG2TZ4EIRBGL4TLOLBMYOAZ4KLIQSZOXKEOWAJBQQJLWSA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299794709 (469h11m39.170877744s) [noise=true , address=145.79.227.37:4567 , piece_id=TZ4C57ISUI3CB3VOLTSM5W5QVDI7PVXV67PFJI3ZSVSXHTKPYPSA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.300142221 (469h11m39.171225256s) [noise=true , address=145.79.227.8:4331 , piece_id=ELLVNJFOVY5HSAHZDQYGQVO7OYHFZNFHCBJKHQFV5K6EV2M3JTRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.300506354 (469h11m39.171589389s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.301142795 (469h11m39.17222584s) [noise=true , address=145.79.227.36:4517 , piece_id=HBQU7CMAO2KJS6ODW2WHICBAWQSH6TQQPOQSENKDEISWCWFN2CMA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.301625835 (469h11m39.17270891s) [noise=true , address=145.79.227.45:4229 , piece_id=GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.302170964 (469h11m39.173254009s) [noise=true , address=145.79.227.3:4150 , piece_id=OHGA4EA33AVEDCJLB6QSUJLD3W2TV6YNN6FBVFFKTTSF2B6XDPRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.302629947 (469h11m39.173712982s) [noise=true , address=145.79.227.19:4079 , piece_id=ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303079704 (469h11m39.174162749s) [noise=true , address=145.79.227.15:4875 , piece_id=FUMLT2ODHUE3ETSNWVKBYLSJXYVCHUW5KAJJE5AECNL3IAQYYCLQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303499804 (469h11m39.174582849s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303921386 (469h11m39.175004432s) [noise=true , address=145.79.227.47:4265 , piece_id=TVUXNR6K35A3CWFDN4JRKM4Q6RGBC4LP2XKPXJFN2AJL32ESEBPA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:41.172453460 (469h11m41.043536495s) [upload=411834]: Next returned nil segment Dec 25 04:40:41.172473798 (469h11m41.043556833s) [upload=411834]: Waiting for error group managing segments... Dec 25 04:40:41.483361556 (469h11m41.354444601s) [noise=true , address=145.79.227.37:4567 , piece_id=TZ4C57ISUI3CB3VOLTSM5W5QVDI7PVXV67PFJI3ZSVSXHTKPYPSA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.484283170 (469h11m41.355366205s) [noise=true , address=145.79.227.5:4221 , piece_id=C4SXUVTG2TZ4EIRBGL4TLOLBMYOAZ4KLIQSZOXKEOWAJBQQJLWSA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.515393699 (469h11m41.386476744s) [noise=true , address=145.79.227.51:4411 , piece_id=WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.526179387 (469h11m41.397262422s) [noise=true , address=45.91.116.5:4300 , piece_id=NTWC3RWYQQBBKPMXGUG44R2B5GWC2Z4LDFU3GTWOWCT3JIRIZOOQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.642380892 (469h11m41.513463937s) [noise=true , address=145.79.227.36:4517 , piece_id=HBQU7CMAO2KJS6ODW2WHICBAWQSH6TQQPOQSENKDEISWCWFN2CMA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.662326662 (469h11m41.533409707s) [noise=true , address=145.79.227.3:4150 , piece_id=OHGA4EA33AVEDCJLB6QSUJLD3W2TV6YNN6FBVFFKTTSF2B6XDPRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.758852130 (469h11m41.629935175s) [noise=true , address=145.79.227.47:4265 , piece_id=TVUXNR6K35A3CWFDN4JRKM4Q6RGBC4LP2XKPXJFN2AJL32ESEBPA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.823449082 (469h11m41.694532117s) [noise=true , address=145.79.227.17:4970 , piece_id=6DPKJ762IUN4A7UEZXLDERCVDW24QEKGEHFETAYZOKWQPVN2QJYA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.952636616 (469h11m41.823719661s) [noise=true , address=145.79.227.19:4079 , piece_id=ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.025177123 (469h11m41.896260168s) [noise=true , address=145.79.227.45:4229 , piece_id=GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.222526274 (469h11m42.093609319s) [noise=true , address=145.79.227.8:4331 , piece_id=ELLVNJFOVY5HSAHZDQYGQVO7OYHFZNFHCBJKHQFV5K6EV2M3JTRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.250024374 (469h11m42.121107409s) [noise=true , address=145.79.227.2:4114 , piece_id=5ISXZKQSH4CLLTF3ILGAXHV6STMLRUSZP6UATU2677X2FUPLBCRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.651787921 (469h11m42.522870966s) [noise=true , address=145.79.227.15:4875 , piece_id=FUMLT2ODHUE3ETSNWVKBYLSJXYVCHUW5KAJJE5AECNL3IAQYYCLQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.651814982 (469h11m42.522898027s) [seg_pos={0 0}, upload=411834]: Segment reached optimal threshold of 13 pieces. Dec 25 04:40:42.652052093 (469h11m42.523135129s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: write tcp 188.166.225.170:36650->77.90.31.15:4697: use of closed network connection; storage node overloaded, request limit: 2000 Dec 25 04:40:42.652069537 (469h11m42.523152572s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652177673 (469h11m42.523260708s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd): protocol: write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection; write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection Dec 25 04:40:42.652184456 (469h11m42.523267501s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652230243 (469h11m42.523313308s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq): protocol: write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection Dec 25 04:40:42.652240693 (469h11m42.523323778s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652268265 (469h11m42.52335131s) [seg_pos={0 0}, upload=411834]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 04:40:42.652354770 (469h11m42.523437815s) [upload=411834]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 04:40:42.652361693 (469h11m42.523444738s) [upload=411834]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 04:40:42.959704440 (469h11m42.830787485s) [upload=411834]: Done uploading segments... Dec 25 04:40:42.959728234 (469h11m42.83081127s) [upload=411834]: Upload finished. err: <nil> Duration: 5.684781414s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..................................................X.........] duration: 3.673851451s trace: https://jaeger.dev.storj.io/trace/835f8cfb18d0547 FAIL (2025-12-25 05:22:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.673868974s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...................................................X........] duration: 3.659910944s trace: https://jaeger.dev.storj.io/trace/835f8cfb18d0547 FAIL (2025-12-25 05:21:27 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.659930982s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 5.886221727s trace: https://jaeger.dev.storj.io/trace/681cf842d2260d41 FAIL (2025-12-25 05:21:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, Piece ID: ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ) context deadline exceeded; piecestore: (Node ID: 1DT68m6J4PpDk3XUDhDaaqV46NLKVz4A23716JLFxEAh122GeR, Piece ID: WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA) context deadline exceeded; piecestore: (Node ID: 12wWYnhWHSqqAFgje4p9Cr7WYaJ6dJWsCqvrx8ZqVFoakjuFeka, Piece ID: GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q) context deadline exceeded ======LOGS====== nline: false Dec 25 04:40:39.039085560 (469h11m38.910168605s) [upload=411834]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 04:40:39.039095028 (469h11m38.910178073s) [upload=411834]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 04:40:39.297407613 (469h11m39.168490658s) [seg_pos={0 0}, upload=411834]: Begin upload segment... Dec 25 04:40:39.297471084 (469h11m39.168554119s) [seg_pos={0 0}, upload=411834]: Done begin upload segment. Dec 25 04:40:39.297511581 (469h11m39.168594616s) [noise=true , address=145.79.227.51:4411 , piece_id=WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.297663310 (469h11m39.168746345s) [noise=true , address=145.79.227.17:4970 , piece_id=6DPKJ762IUN4A7UEZXLDERCVDW24QEKGEHFETAYZOKWQPVN2QJYA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.298405574 (469h11m39.169488619s) [noise=true , address=145.79.227.2:4114 , piece_id=5ISXZKQSH4CLLTF3ILGAXHV6STMLRUSZP6UATU2677X2FUPLBCRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.298797961 (469h11m39.169881006s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299124392 (469h11m39.170207437s) [noise=true , address=45.91.116.5:4300 , piece_id=NTWC3RWYQQBBKPMXGUG44R2B5GWC2Z4LDFU3GTWOWCT3JIRIZOOQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299438650 (469h11m39.170521685s) [noise=true , address=145.79.227.5:4221 , piece_id=C4SXUVTG2TZ4EIRBGL4TLOLBMYOAZ4KLIQSZOXKEOWAJBQQJLWSA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299794709 (469h11m39.170877744s) [noise=true , address=145.79.227.37:4567 , piece_id=TZ4C57ISUI3CB3VOLTSM5W5QVDI7PVXV67PFJI3ZSVSXHTKPYPSA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.300142221 (469h11m39.171225256s) [noise=true , address=145.79.227.8:4331 , piece_id=ELLVNJFOVY5HSAHZDQYGQVO7OYHFZNFHCBJKHQFV5K6EV2M3JTRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.300506354 (469h11m39.171589389s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.301142795 (469h11m39.17222584s) [noise=true , address=145.79.227.36:4517 , piece_id=HBQU7CMAO2KJS6ODW2WHICBAWQSH6TQQPOQSENKDEISWCWFN2CMA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.301625835 (469h11m39.17270891s) [noise=true , address=145.79.227.45:4229 , piece_id=GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.302170964 (469h11m39.173254009s) [noise=true , address=145.79.227.3:4150 , piece_id=OHGA4EA33AVEDCJLB6QSUJLD3W2TV6YNN6FBVFFKTTSF2B6XDPRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.302629947 (469h11m39.173712982s) [noise=true , address=145.79.227.19:4079 , piece_id=ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303079704 (469h11m39.174162749s) [noise=true , address=145.79.227.15:4875 , piece_id=FUMLT2ODHUE3ETSNWVKBYLSJXYVCHUW5KAJJE5AECNL3IAQYYCLQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303499804 (469h11m39.174582849s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303921386 (469h11m39.175004432s) [noise=true , address=145.79.227.47:4265 , piece_id=TVUXNR6K35A3CWFDN4JRKM4Q6RGBC4LP2XKPXJFN2AJL32ESEBPA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:41.172453460 (469h11m41.043536495s) [upload=411834]: Next returned nil segment Dec 25 04:40:41.172473798 (469h11m41.043556833s) [upload=411834]: Waiting for error group managing segments... Dec 25 04:40:41.483361556 (469h11m41.354444601s) [noise=true , address=145.79.227.37:4567 , piece_id=TZ4C57ISUI3CB3VOLTSM5W5QVDI7PVXV67PFJI3ZSVSXHTKPYPSA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.484283170 (469h11m41.355366205s) [noise=true , address=145.79.227.5:4221 , piece_id=C4SXUVTG2TZ4EIRBGL4TLOLBMYOAZ4KLIQSZOXKEOWAJBQQJLWSA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.515393699 (469h11m41.386476744s) [noise=true , address=145.79.227.51:4411 , piece_id=WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.526179387 (469h11m41.397262422s) [noise=true , address=45.91.116.5:4300 , piece_id=NTWC3RWYQQBBKPMXGUG44R2B5GWC2Z4LDFU3GTWOWCT3JIRIZOOQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.642380892 (469h11m41.513463937s) [noise=true , address=145.79.227.36:4517 , piece_id=HBQU7CMAO2KJS6ODW2WHICBAWQSH6TQQPOQSENKDEISWCWFN2CMA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.662326662 (469h11m41.533409707s) [noise=true , address=145.79.227.3:4150 , piece_id=OHGA4EA33AVEDCJLB6QSUJLD3W2TV6YNN6FBVFFKTTSF2B6XDPRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.758852130 (469h11m41.629935175s) [noise=true , address=145.79.227.47:4265 , piece_id=TVUXNR6K35A3CWFDN4JRKM4Q6RGBC4LP2XKPXJFN2AJL32ESEBPA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.823449082 (469h11m41.694532117s) [noise=true , address=145.79.227.17:4970 , piece_id=6DPKJ762IUN4A7UEZXLDERCVDW24QEKGEHFETAYZOKWQPVN2QJYA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.952636616 (469h11m41.823719661s) [noise=true , address=145.79.227.19:4079 , piece_id=ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.025177123 (469h11m41.896260168s) [noise=true , address=145.79.227.45:4229 , piece_id=GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.222526274 (469h11m42.093609319s) [noise=true , address=145.79.227.8:4331 , piece_id=ELLVNJFOVY5HSAHZDQYGQVO7OYHFZNFHCBJKHQFV5K6EV2M3JTRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.250024374 (469h11m42.121107409s) [noise=true , address=145.79.227.2:4114 , piece_id=5ISXZKQSH4CLLTF3ILGAXHV6STMLRUSZP6UATU2677X2FUPLBCRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.651787921 (469h11m42.522870966s) [noise=true , address=145.79.227.15:4875 , piece_id=FUMLT2ODHUE3ETSNWVKBYLSJXYVCHUW5KAJJE5AECNL3IAQYYCLQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.651814982 (469h11m42.522898027s) [seg_pos={0 0}, upload=411834]: Segment reached optimal threshold of 13 pieces. Dec 25 04:40:42.652052093 (469h11m42.523135129s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: write tcp 188.166.225.170:36650->77.90.31.15:4697: use of closed network connection; storage node overloaded, request limit: 2000 Dec 25 04:40:42.652069537 (469h11m42.523152572s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652177673 (469h11m42.523260708s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd): protocol: write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection; write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection Dec 25 04:40:42.652184456 (469h11m42.523267501s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652230243 (469h11m42.523313308s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq): protocol: write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection Dec 25 04:40:42.652240693 (469h11m42.523323778s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652268265 (469h11m42.52335131s) [seg_pos={0 0}, upload=411834]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 04:40:42.652354770 (469h11m42.523437815s) [upload=411834]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 04:40:42.652361693 (469h11m42.523444738s) [upload=411834]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 04:40:42.959704440 (469h11m42.830787485s) [upload=411834]: Done uploading segments... Dec 25 04:40:42.959728234 (469h11m42.83081127s) [upload=411834]: Upload finished. err: <nil> Duration: 5.886249089s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 5.945891217s trace: https://jaeger.dev.storj.io/trace/681cf842d2260d41 FAIL (2025-12-25 05:21:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, Piece ID: ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ) context deadline exceeded; piecestore: (Node ID: 1DT68m6J4PpDk3XUDhDaaqV46NLKVz4A23716JLFxEAh122GeR, Piece ID: WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA) context deadline exceeded; piecestore: (Node ID: 12wWYnhWHSqqAFgje4p9Cr7WYaJ6dJWsCqvrx8ZqVFoakjuFeka, Piece ID: GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q) context deadline exceeded ======LOGS====== nline: false Dec 25 04:40:39.039085560 (469h11m38.910168605s) [upload=411834]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 04:40:39.039095028 (469h11m38.910178073s) [upload=411834]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 04:40:39.297407613 (469h11m39.168490658s) [seg_pos={0 0}, upload=411834]: Begin upload segment... Dec 25 04:40:39.297471084 (469h11m39.168554119s) [seg_pos={0 0}, upload=411834]: Done begin upload segment. Dec 25 04:40:39.297511581 (469h11m39.168594616s) [noise=true , address=145.79.227.51:4411 , piece_id=WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.297663310 (469h11m39.168746345s) [noise=true , address=145.79.227.17:4970 , piece_id=6DPKJ762IUN4A7UEZXLDERCVDW24QEKGEHFETAYZOKWQPVN2QJYA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.298405574 (469h11m39.169488619s) [noise=true , address=145.79.227.2:4114 , piece_id=5ISXZKQSH4CLLTF3ILGAXHV6STMLRUSZP6UATU2677X2FUPLBCRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.298797961 (469h11m39.169881006s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299124392 (469h11m39.170207437s) [noise=true , address=45.91.116.5:4300 , piece_id=NTWC3RWYQQBBKPMXGUG44R2B5GWC2Z4LDFU3GTWOWCT3JIRIZOOQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299438650 (469h11m39.170521685s) [noise=true , address=145.79.227.5:4221 , piece_id=C4SXUVTG2TZ4EIRBGL4TLOLBMYOAZ4KLIQSZOXKEOWAJBQQJLWSA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299794709 (469h11m39.170877744s) [noise=true , address=145.79.227.37:4567 , piece_id=TZ4C57ISUI3CB3VOLTSM5W5QVDI7PVXV67PFJI3ZSVSXHTKPYPSA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.300142221 (469h11m39.171225256s) [noise=true , address=145.79.227.8:4331 , piece_id=ELLVNJFOVY5HSAHZDQYGQVO7OYHFZNFHCBJKHQFV5K6EV2M3JTRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.300506354 (469h11m39.171589389s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.301142795 (469h11m39.17222584s) [noise=true , address=145.79.227.36:4517 , piece_id=HBQU7CMAO2KJS6ODW2WHICBAWQSH6TQQPOQSENKDEISWCWFN2CMA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.301625835 (469h11m39.17270891s) [noise=true , address=145.79.227.45:4229 , piece_id=GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.302170964 (469h11m39.173254009s) [noise=true , address=145.79.227.3:4150 , piece_id=OHGA4EA33AVEDCJLB6QSUJLD3W2TV6YNN6FBVFFKTTSF2B6XDPRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.302629947 (469h11m39.173712982s) [noise=true , address=145.79.227.19:4079 , piece_id=ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303079704 (469h11m39.174162749s) [noise=true , address=145.79.227.15:4875 , piece_id=FUMLT2ODHUE3ETSNWVKBYLSJXYVCHUW5KAJJE5AECNL3IAQYYCLQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303499804 (469h11m39.174582849s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303921386 (469h11m39.175004432s) [noise=true , address=145.79.227.47:4265 , piece_id=TVUXNR6K35A3CWFDN4JRKM4Q6RGBC4LP2XKPXJFN2AJL32ESEBPA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:41.172453460 (469h11m41.043536495s) [upload=411834]: Next returned nil segment Dec 25 04:40:41.172473798 (469h11m41.043556833s) [upload=411834]: Waiting for error group managing segments... Dec 25 04:40:41.483361556 (469h11m41.354444601s) [noise=true , address=145.79.227.37:4567 , piece_id=TZ4C57ISUI3CB3VOLTSM5W5QVDI7PVXV67PFJI3ZSVSXHTKPYPSA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.484283170 (469h11m41.355366205s) [noise=true , address=145.79.227.5:4221 , piece_id=C4SXUVTG2TZ4EIRBGL4TLOLBMYOAZ4KLIQSZOXKEOWAJBQQJLWSA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.515393699 (469h11m41.386476744s) [noise=true , address=145.79.227.51:4411 , piece_id=WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.526179387 (469h11m41.397262422s) [noise=true , address=45.91.116.5:4300 , piece_id=NTWC3RWYQQBBKPMXGUG44R2B5GWC2Z4LDFU3GTWOWCT3JIRIZOOQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.642380892 (469h11m41.513463937s) [noise=true , address=145.79.227.36:4517 , piece_id=HBQU7CMAO2KJS6ODW2WHICBAWQSH6TQQPOQSENKDEISWCWFN2CMA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.662326662 (469h11m41.533409707s) [noise=true , address=145.79.227.3:4150 , piece_id=OHGA4EA33AVEDCJLB6QSUJLD3W2TV6YNN6FBVFFKTTSF2B6XDPRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.758852130 (469h11m41.629935175s) [noise=true , address=145.79.227.47:4265 , piece_id=TVUXNR6K35A3CWFDN4JRKM4Q6RGBC4LP2XKPXJFN2AJL32ESEBPA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.823449082 (469h11m41.694532117s) [noise=true , address=145.79.227.17:4970 , piece_id=6DPKJ762IUN4A7UEZXLDERCVDW24QEKGEHFETAYZOKWQPVN2QJYA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.952636616 (469h11m41.823719661s) [noise=true , address=145.79.227.19:4079 , piece_id=ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.025177123 (469h11m41.896260168s) [noise=true , address=145.79.227.45:4229 , piece_id=GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.222526274 (469h11m42.093609319s) [noise=true , address=145.79.227.8:4331 , piece_id=ELLVNJFOVY5HSAHZDQYGQVO7OYHFZNFHCBJKHQFV5K6EV2M3JTRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.250024374 (469h11m42.121107409s) [noise=true , address=145.79.227.2:4114 , piece_id=5ISXZKQSH4CLLTF3ILGAXHV6STMLRUSZP6UATU2677X2FUPLBCRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.651787921 (469h11m42.522870966s) [noise=true , address=145.79.227.15:4875 , piece_id=FUMLT2ODHUE3ETSNWVKBYLSJXYVCHUW5KAJJE5AECNL3IAQYYCLQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.651814982 (469h11m42.522898027s) [seg_pos={0 0}, upload=411834]: Segment reached optimal threshold of 13 pieces. Dec 25 04:40:42.652052093 (469h11m42.523135129s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: write tcp 188.166.225.170:36650->77.90.31.15:4697: use of closed network connection; storage node overloaded, request limit: 2000 Dec 25 04:40:42.652069537 (469h11m42.523152572s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652177673 (469h11m42.523260708s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd): protocol: write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection; write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection Dec 25 04:40:42.652184456 (469h11m42.523267501s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652230243 (469h11m42.523313308s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq): protocol: write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection Dec 25 04:40:42.652240693 (469h11m42.523323778s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652268265 (469h11m42.52335131s) [seg_pos={0 0}, upload=411834]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 04:40:42.652354770 (469h11m42.523437815s) [upload=411834]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 04:40:42.652361693 (469h11m42.523444738s) [upload=411834]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 04:40:42.959704440 (469h11m42.830787485s) [upload=411834]: Done uploading segments... Dec 25 04:40:42.959728234 (469h11m42.83081127s) [upload=411834]: Upload finished. err: <nil> Duration: 5.945918648s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....................................................X.......] duration: 3.466450306s trace: https://jaeger.dev.storj.io/trace/835f8cfb18d0547 FAIL (2025-12-25 05:20:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.466468222s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 5.925326695s trace: https://jaeger.dev.storj.io/trace/681cf842d2260d41 FAIL (2025-12-25 05:20:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, Piece ID: ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ) context deadline exceeded; piecestore: (Node ID: 1DT68m6J4PpDk3XUDhDaaqV46NLKVz4A23716JLFxEAh122GeR, Piece ID: WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA) context deadline exceeded; piecestore: (Node ID: 12wWYnhWHSqqAFgje4p9Cr7WYaJ6dJWsCqvrx8ZqVFoakjuFeka, Piece ID: GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q) context deadline exceeded ======LOGS====== nline: false Dec 25 04:40:39.039085560 (469h11m38.910168605s) [upload=411834]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 04:40:39.039095028 (469h11m38.910178073s) [upload=411834]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 04:40:39.297407613 (469h11m39.168490658s) [seg_pos={0 0}, upload=411834]: Begin upload segment... Dec 25 04:40:39.297471084 (469h11m39.168554119s) [seg_pos={0 0}, upload=411834]: Done begin upload segment. Dec 25 04:40:39.297511581 (469h11m39.168594616s) [noise=true , address=145.79.227.51:4411 , piece_id=WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.297663310 (469h11m39.168746345s) [noise=true , address=145.79.227.17:4970 , piece_id=6DPKJ762IUN4A7UEZXLDERCVDW24QEKGEHFETAYZOKWQPVN2QJYA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.298405574 (469h11m39.169488619s) [noise=true , address=145.79.227.2:4114 , piece_id=5ISXZKQSH4CLLTF3ILGAXHV6STMLRUSZP6UATU2677X2FUPLBCRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.298797961 (469h11m39.169881006s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299124392 (469h11m39.170207437s) [noise=true , address=45.91.116.5:4300 , piece_id=NTWC3RWYQQBBKPMXGUG44R2B5GWC2Z4LDFU3GTWOWCT3JIRIZOOQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299438650 (469h11m39.170521685s) [noise=true , address=145.79.227.5:4221 , piece_id=C4SXUVTG2TZ4EIRBGL4TLOLBMYOAZ4KLIQSZOXKEOWAJBQQJLWSA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299794709 (469h11m39.170877744s) [noise=true , address=145.79.227.37:4567 , piece_id=TZ4C57ISUI3CB3VOLTSM5W5QVDI7PVXV67PFJI3ZSVSXHTKPYPSA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.300142221 (469h11m39.171225256s) [noise=true , address=145.79.227.8:4331 , piece_id=ELLVNJFOVY5HSAHZDQYGQVO7OYHFZNFHCBJKHQFV5K6EV2M3JTRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.300506354 (469h11m39.171589389s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.301142795 (469h11m39.17222584s) [noise=true , address=145.79.227.36:4517 , piece_id=HBQU7CMAO2KJS6ODW2WHICBAWQSH6TQQPOQSENKDEISWCWFN2CMA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.301625835 (469h11m39.17270891s) [noise=true , address=145.79.227.45:4229 , piece_id=GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.302170964 (469h11m39.173254009s) [noise=true , address=145.79.227.3:4150 , piece_id=OHGA4EA33AVEDCJLB6QSUJLD3W2TV6YNN6FBVFFKTTSF2B6XDPRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.302629947 (469h11m39.173712982s) [noise=true , address=145.79.227.19:4079 , piece_id=ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303079704 (469h11m39.174162749s) [noise=true , address=145.79.227.15:4875 , piece_id=FUMLT2ODHUE3ETSNWVKBYLSJXYVCHUW5KAJJE5AECNL3IAQYYCLQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303499804 (469h11m39.174582849s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303921386 (469h11m39.175004432s) [noise=true , address=145.79.227.47:4265 , piece_id=TVUXNR6K35A3CWFDN4JRKM4Q6RGBC4LP2XKPXJFN2AJL32ESEBPA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:41.172453460 (469h11m41.043536495s) [upload=411834]: Next returned nil segment Dec 25 04:40:41.172473798 (469h11m41.043556833s) [upload=411834]: Waiting for error group managing segments... Dec 25 04:40:41.483361556 (469h11m41.354444601s) [noise=true , address=145.79.227.37:4567 , piece_id=TZ4C57ISUI3CB3VOLTSM5W5QVDI7PVXV67PFJI3ZSVSXHTKPYPSA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.484283170 (469h11m41.355366205s) [noise=true , address=145.79.227.5:4221 , piece_id=C4SXUVTG2TZ4EIRBGL4TLOLBMYOAZ4KLIQSZOXKEOWAJBQQJLWSA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.515393699 (469h11m41.386476744s) [noise=true , address=145.79.227.51:4411 , piece_id=WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.526179387 (469h11m41.397262422s) [noise=true , address=45.91.116.5:4300 , piece_id=NTWC3RWYQQBBKPMXGUG44R2B5GWC2Z4LDFU3GTWOWCT3JIRIZOOQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.642380892 (469h11m41.513463937s) [noise=true , address=145.79.227.36:4517 , piece_id=HBQU7CMAO2KJS6ODW2WHICBAWQSH6TQQPOQSENKDEISWCWFN2CMA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.662326662 (469h11m41.533409707s) [noise=true , address=145.79.227.3:4150 , piece_id=OHGA4EA33AVEDCJLB6QSUJLD3W2TV6YNN6FBVFFKTTSF2B6XDPRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.758852130 (469h11m41.629935175s) [noise=true , address=145.79.227.47:4265 , piece_id=TVUXNR6K35A3CWFDN4JRKM4Q6RGBC4LP2XKPXJFN2AJL32ESEBPA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.823449082 (469h11m41.694532117s) [noise=true , address=145.79.227.17:4970 , piece_id=6DPKJ762IUN4A7UEZXLDERCVDW24QEKGEHFETAYZOKWQPVN2QJYA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.952636616 (469h11m41.823719661s) [noise=true , address=145.79.227.19:4079 , piece_id=ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.025177123 (469h11m41.896260168s) [noise=true , address=145.79.227.45:4229 , piece_id=GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.222526274 (469h11m42.093609319s) [noise=true , address=145.79.227.8:4331 , piece_id=ELLVNJFOVY5HSAHZDQYGQVO7OYHFZNFHCBJKHQFV5K6EV2M3JTRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.250024374 (469h11m42.121107409s) [noise=true , address=145.79.227.2:4114 , piece_id=5ISXZKQSH4CLLTF3ILGAXHV6STMLRUSZP6UATU2677X2FUPLBCRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.651787921 (469h11m42.522870966s) [noise=true , address=145.79.227.15:4875 , piece_id=FUMLT2ODHUE3ETSNWVKBYLSJXYVCHUW5KAJJE5AECNL3IAQYYCLQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.651814982 (469h11m42.522898027s) [seg_pos={0 0}, upload=411834]: Segment reached optimal threshold of 13 pieces. Dec 25 04:40:42.652052093 (469h11m42.523135129s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: write tcp 188.166.225.170:36650->77.90.31.15:4697: use of closed network connection; storage node overloaded, request limit: 2000 Dec 25 04:40:42.652069537 (469h11m42.523152572s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652177673 (469h11m42.523260708s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd): protocol: write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection; write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection Dec 25 04:40:42.652184456 (469h11m42.523267501s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652230243 (469h11m42.523313308s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq): protocol: write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection Dec 25 04:40:42.652240693 (469h11m42.523323778s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652268265 (469h11m42.52335131s) [seg_pos={0 0}, upload=411834]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 04:40:42.652354770 (469h11m42.523437815s) [upload=411834]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 04:40:42.652361693 (469h11m42.523444738s) [upload=411834]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 04:40:42.959704440 (469h11m42.830787485s) [upload=411834]: Done uploading segments... Dec 25 04:40:42.959728234 (469h11m42.83081127s) [upload=411834]: Upload finished. err: <nil> Duration: 5.925358445s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 6.292541643s trace: https://jaeger.dev.storj.io/trace/681cf842d2260d41 FAIL (2025-12-25 05:20:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, Piece ID: ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ) context deadline exceeded; piecestore: (Node ID: 1DT68m6J4PpDk3XUDhDaaqV46NLKVz4A23716JLFxEAh122GeR, Piece ID: WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA) context deadline exceeded; piecestore: (Node ID: 12wWYnhWHSqqAFgje4p9Cr7WYaJ6dJWsCqvrx8ZqVFoakjuFeka, Piece ID: GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q) context deadline exceeded ======LOGS====== nline: false Dec 25 04:40:39.039085560 (469h11m38.910168605s) [upload=411834]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 04:40:39.039095028 (469h11m38.910178073s) [upload=411834]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 04:40:39.297407613 (469h11m39.168490658s) [seg_pos={0 0}, upload=411834]: Begin upload segment... Dec 25 04:40:39.297471084 (469h11m39.168554119s) [seg_pos={0 0}, upload=411834]: Done begin upload segment. Dec 25 04:40:39.297511581 (469h11m39.168594616s) [noise=true , address=145.79.227.51:4411 , piece_id=WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.297663310 (469h11m39.168746345s) [noise=true , address=145.79.227.17:4970 , piece_id=6DPKJ762IUN4A7UEZXLDERCVDW24QEKGEHFETAYZOKWQPVN2QJYA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.298405574 (469h11m39.169488619s) [noise=true , address=145.79.227.2:4114 , piece_id=5ISXZKQSH4CLLTF3ILGAXHV6STMLRUSZP6UATU2677X2FUPLBCRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.298797961 (469h11m39.169881006s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299124392 (469h11m39.170207437s) [noise=true , address=45.91.116.5:4300 , piece_id=NTWC3RWYQQBBKPMXGUG44R2B5GWC2Z4LDFU3GTWOWCT3JIRIZOOQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299438650 (469h11m39.170521685s) [noise=true , address=145.79.227.5:4221 , piece_id=C4SXUVTG2TZ4EIRBGL4TLOLBMYOAZ4KLIQSZOXKEOWAJBQQJLWSA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299794709 (469h11m39.170877744s) [noise=true , address=145.79.227.37:4567 , piece_id=TZ4C57ISUI3CB3VOLTSM5W5QVDI7PVXV67PFJI3ZSVSXHTKPYPSA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.300142221 (469h11m39.171225256s) [noise=true , address=145.79.227.8:4331 , piece_id=ELLVNJFOVY5HSAHZDQYGQVO7OYHFZNFHCBJKHQFV5K6EV2M3JTRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.300506354 (469h11m39.171589389s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.301142795 (469h11m39.17222584s) [noise=true , address=145.79.227.36:4517 , piece_id=HBQU7CMAO2KJS6ODW2WHICBAWQSH6TQQPOQSENKDEISWCWFN2CMA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.301625835 (469h11m39.17270891s) [noise=true , address=145.79.227.45:4229 , piece_id=GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.302170964 (469h11m39.173254009s) [noise=true , address=145.79.227.3:4150 , piece_id=OHGA4EA33AVEDCJLB6QSUJLD3W2TV6YNN6FBVFFKTTSF2B6XDPRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.302629947 (469h11m39.173712982s) [noise=true , address=145.79.227.19:4079 , piece_id=ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303079704 (469h11m39.174162749s) [noise=true , address=145.79.227.15:4875 , piece_id=FUMLT2ODHUE3ETSNWVKBYLSJXYVCHUW5KAJJE5AECNL3IAQYYCLQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303499804 (469h11m39.174582849s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303921386 (469h11m39.175004432s) [noise=true , address=145.79.227.47:4265 , piece_id=TVUXNR6K35A3CWFDN4JRKM4Q6RGBC4LP2XKPXJFN2AJL32ESEBPA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:41.172453460 (469h11m41.043536495s) [upload=411834]: Next returned nil segment Dec 25 04:40:41.172473798 (469h11m41.043556833s) [upload=411834]: Waiting for error group managing segments... Dec 25 04:40:41.483361556 (469h11m41.354444601s) [noise=true , address=145.79.227.37:4567 , piece_id=TZ4C57ISUI3CB3VOLTSM5W5QVDI7PVXV67PFJI3ZSVSXHTKPYPSA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.484283170 (469h11m41.355366205s) [noise=true , address=145.79.227.5:4221 , piece_id=C4SXUVTG2TZ4EIRBGL4TLOLBMYOAZ4KLIQSZOXKEOWAJBQQJLWSA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.515393699 (469h11m41.386476744s) [noise=true , address=145.79.227.51:4411 , piece_id=WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.526179387 (469h11m41.397262422s) [noise=true , address=45.91.116.5:4300 , piece_id=NTWC3RWYQQBBKPMXGUG44R2B5GWC2Z4LDFU3GTWOWCT3JIRIZOOQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.642380892 (469h11m41.513463937s) [noise=true , address=145.79.227.36:4517 , piece_id=HBQU7CMAO2KJS6ODW2WHICBAWQSH6TQQPOQSENKDEISWCWFN2CMA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.662326662 (469h11m41.533409707s) [noise=true , address=145.79.227.3:4150 , piece_id=OHGA4EA33AVEDCJLB6QSUJLD3W2TV6YNN6FBVFFKTTSF2B6XDPRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.758852130 (469h11m41.629935175s) [noise=true , address=145.79.227.47:4265 , piece_id=TVUXNR6K35A3CWFDN4JRKM4Q6RGBC4LP2XKPXJFN2AJL32ESEBPA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.823449082 (469h11m41.694532117s) [noise=true , address=145.79.227.17:4970 , piece_id=6DPKJ762IUN4A7UEZXLDERCVDW24QEKGEHFETAYZOKWQPVN2QJYA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.952636616 (469h11m41.823719661s) [noise=true , address=145.79.227.19:4079 , piece_id=ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.025177123 (469h11m41.896260168s) [noise=true , address=145.79.227.45:4229 , piece_id=GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.222526274 (469h11m42.093609319s) [noise=true , address=145.79.227.8:4331 , piece_id=ELLVNJFOVY5HSAHZDQYGQVO7OYHFZNFHCBJKHQFV5K6EV2M3JTRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.250024374 (469h11m42.121107409s) [noise=true , address=145.79.227.2:4114 , piece_id=5ISXZKQSH4CLLTF3ILGAXHV6STMLRUSZP6UATU2677X2FUPLBCRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.651787921 (469h11m42.522870966s) [noise=true , address=145.79.227.15:4875 , piece_id=FUMLT2ODHUE3ETSNWVKBYLSJXYVCHUW5KAJJE5AECNL3IAQYYCLQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.651814982 (469h11m42.522898027s) [seg_pos={0 0}, upload=411834]: Segment reached optimal threshold of 13 pieces. Dec 25 04:40:42.652052093 (469h11m42.523135129s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: write tcp 188.166.225.170:36650->77.90.31.15:4697: use of closed network connection; storage node overloaded, request limit: 2000 Dec 25 04:40:42.652069537 (469h11m42.523152572s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652177673 (469h11m42.523260708s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd): protocol: write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection; write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection Dec 25 04:40:42.652184456 (469h11m42.523267501s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652230243 (469h11m42.523313308s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq): protocol: write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection Dec 25 04:40:42.652240693 (469h11m42.523323778s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652268265 (469h11m42.52335131s) [seg_pos={0 0}, upload=411834]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 04:40:42.652354770 (469h11m42.523437815s) [upload=411834]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 04:40:42.652361693 (469h11m42.523444738s) [upload=411834]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 04:40:42.959704440 (469h11m42.830787485s) [upload=411834]: Done uploading segments... Dec 25 04:40:42.959728234 (469h11m42.83081127s) [upload=411834]: Upload finished. err: <nil> Duration: 6.292573763s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....................................................X......] duration: 3.699044905s trace: https://jaeger.dev.storj.io/trace/835f8cfb18d0547 FAIL (2025-12-25 05:19:34 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.699070725s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 5.693178245s trace: https://jaeger.dev.storj.io/trace/681cf842d2260d41 FAIL (2025-12-25 05:19:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, Piece ID: ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ) context deadline exceeded; piecestore: (Node ID: 1DT68m6J4PpDk3XUDhDaaqV46NLKVz4A23716JLFxEAh122GeR, Piece ID: WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA) context deadline exceeded; piecestore: (Node ID: 12wWYnhWHSqqAFgje4p9Cr7WYaJ6dJWsCqvrx8ZqVFoakjuFeka, Piece ID: GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q) context deadline exceeded ======LOGS====== nline: false Dec 25 04:40:39.039085560 (469h11m38.910168605s) [upload=411834]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 04:40:39.039095028 (469h11m38.910178073s) [upload=411834]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 04:40:39.297407613 (469h11m39.168490658s) [seg_pos={0 0}, upload=411834]: Begin upload segment... Dec 25 04:40:39.297471084 (469h11m39.168554119s) [seg_pos={0 0}, upload=411834]: Done begin upload segment. Dec 25 04:40:39.297511581 (469h11m39.168594616s) [noise=true , address=145.79.227.51:4411 , piece_id=WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.297663310 (469h11m39.168746345s) [noise=true , address=145.79.227.17:4970 , piece_id=6DPKJ762IUN4A7UEZXLDERCVDW24QEKGEHFETAYZOKWQPVN2QJYA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.298405574 (469h11m39.169488619s) [noise=true , address=145.79.227.2:4114 , piece_id=5ISXZKQSH4CLLTF3ILGAXHV6STMLRUSZP6UATU2677X2FUPLBCRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.298797961 (469h11m39.169881006s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299124392 (469h11m39.170207437s) [noise=true , address=45.91.116.5:4300 , piece_id=NTWC3RWYQQBBKPMXGUG44R2B5GWC2Z4LDFU3GTWOWCT3JIRIZOOQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299438650 (469h11m39.170521685s) [noise=true , address=145.79.227.5:4221 , piece_id=C4SXUVTG2TZ4EIRBGL4TLOLBMYOAZ4KLIQSZOXKEOWAJBQQJLWSA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299794709 (469h11m39.170877744s) [noise=true , address=145.79.227.37:4567 , piece_id=TZ4C57ISUI3CB3VOLTSM5W5QVDI7PVXV67PFJI3ZSVSXHTKPYPSA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.300142221 (469h11m39.171225256s) [noise=true , address=145.79.227.8:4331 , piece_id=ELLVNJFOVY5HSAHZDQYGQVO7OYHFZNFHCBJKHQFV5K6EV2M3JTRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.300506354 (469h11m39.171589389s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.301142795 (469h11m39.17222584s) [noise=true , address=145.79.227.36:4517 , piece_id=HBQU7CMAO2KJS6ODW2WHICBAWQSH6TQQPOQSENKDEISWCWFN2CMA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.301625835 (469h11m39.17270891s) [noise=true , address=145.79.227.45:4229 , piece_id=GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.302170964 (469h11m39.173254009s) [noise=true , address=145.79.227.3:4150 , piece_id=OHGA4EA33AVEDCJLB6QSUJLD3W2TV6YNN6FBVFFKTTSF2B6XDPRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.302629947 (469h11m39.173712982s) [noise=true , address=145.79.227.19:4079 , piece_id=ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303079704 (469h11m39.174162749s) [noise=true , address=145.79.227.15:4875 , piece_id=FUMLT2ODHUE3ETSNWVKBYLSJXYVCHUW5KAJJE5AECNL3IAQYYCLQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303499804 (469h11m39.174582849s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303921386 (469h11m39.175004432s) [noise=true , address=145.79.227.47:4265 , piece_id=TVUXNR6K35A3CWFDN4JRKM4Q6RGBC4LP2XKPXJFN2AJL32ESEBPA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:41.172453460 (469h11m41.043536495s) [upload=411834]: Next returned nil segment Dec 25 04:40:41.172473798 (469h11m41.043556833s) [upload=411834]: Waiting for error group managing segments... Dec 25 04:40:41.483361556 (469h11m41.354444601s) [noise=true , address=145.79.227.37:4567 , piece_id=TZ4C57ISUI3CB3VOLTSM5W5QVDI7PVXV67PFJI3ZSVSXHTKPYPSA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.484283170 (469h11m41.355366205s) [noise=true , address=145.79.227.5:4221 , piece_id=C4SXUVTG2TZ4EIRBGL4TLOLBMYOAZ4KLIQSZOXKEOWAJBQQJLWSA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.515393699 (469h11m41.386476744s) [noise=true , address=145.79.227.51:4411 , piece_id=WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.526179387 (469h11m41.397262422s) [noise=true , address=45.91.116.5:4300 , piece_id=NTWC3RWYQQBBKPMXGUG44R2B5GWC2Z4LDFU3GTWOWCT3JIRIZOOQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.642380892 (469h11m41.513463937s) [noise=true , address=145.79.227.36:4517 , piece_id=HBQU7CMAO2KJS6ODW2WHICBAWQSH6TQQPOQSENKDEISWCWFN2CMA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.662326662 (469h11m41.533409707s) [noise=true , address=145.79.227.3:4150 , piece_id=OHGA4EA33AVEDCJLB6QSUJLD3W2TV6YNN6FBVFFKTTSF2B6XDPRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.758852130 (469h11m41.629935175s) [noise=true , address=145.79.227.47:4265 , piece_id=TVUXNR6K35A3CWFDN4JRKM4Q6RGBC4LP2XKPXJFN2AJL32ESEBPA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.823449082 (469h11m41.694532117s) [noise=true , address=145.79.227.17:4970 , piece_id=6DPKJ762IUN4A7UEZXLDERCVDW24QEKGEHFETAYZOKWQPVN2QJYA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.952636616 (469h11m41.823719661s) [noise=true , address=145.79.227.19:4079 , piece_id=ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.025177123 (469h11m41.896260168s) [noise=true , address=145.79.227.45:4229 , piece_id=GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.222526274 (469h11m42.093609319s) [noise=true , address=145.79.227.8:4331 , piece_id=ELLVNJFOVY5HSAHZDQYGQVO7OYHFZNFHCBJKHQFV5K6EV2M3JTRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.250024374 (469h11m42.121107409s) [noise=true , address=145.79.227.2:4114 , piece_id=5ISXZKQSH4CLLTF3ILGAXHV6STMLRUSZP6UATU2677X2FUPLBCRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.651787921 (469h11m42.522870966s) [noise=true , address=145.79.227.15:4875 , piece_id=FUMLT2ODHUE3ETSNWVKBYLSJXYVCHUW5KAJJE5AECNL3IAQYYCLQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.651814982 (469h11m42.522898027s) [seg_pos={0 0}, upload=411834]: Segment reached optimal threshold of 13 pieces. Dec 25 04:40:42.652052093 (469h11m42.523135129s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: write tcp 188.166.225.170:36650->77.90.31.15:4697: use of closed network connection; storage node overloaded, request limit: 2000 Dec 25 04:40:42.652069537 (469h11m42.523152572s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652177673 (469h11m42.523260708s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd): protocol: write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection; write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection Dec 25 04:40:42.652184456 (469h11m42.523267501s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652230243 (469h11m42.523313308s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq): protocol: write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection Dec 25 04:40:42.652240693 (469h11m42.523323778s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652268265 (469h11m42.52335131s) [seg_pos={0 0}, upload=411834]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 04:40:42.652354770 (469h11m42.523437815s) [upload=411834]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 04:40:42.652361693 (469h11m42.523444738s) [upload=411834]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 04:40:42.959704440 (469h11m42.830787485s) [upload=411834]: Done uploading segments... Dec 25 04:40:42.959728234 (469h11m42.83081127s) [upload=411834]: Upload finished. err: <nil> Duration: 5.693206598s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......................................................X.....] duration: 3.58741438s trace: https://jaeger.dev.storj.io/trace/835f8cfb18d0547 FAIL (2025-12-25 05:18:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.587439047s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 5.941444448s trace: https://jaeger.dev.storj.io/trace/681cf842d2260d41 FAIL (2025-12-25 05:17:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, Piece ID: ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ) context deadline exceeded; piecestore: (Node ID: 1DT68m6J4PpDk3XUDhDaaqV46NLKVz4A23716JLFxEAh122GeR, Piece ID: WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA) context deadline exceeded; piecestore: (Node ID: 12wWYnhWHSqqAFgje4p9Cr7WYaJ6dJWsCqvrx8ZqVFoakjuFeka, Piece ID: GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q) context deadline exceeded ======LOGS====== nline: false Dec 25 04:40:39.039085560 (469h11m38.910168605s) [upload=411834]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 04:40:39.039095028 (469h11m38.910178073s) [upload=411834]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 04:40:39.297407613 (469h11m39.168490658s) [seg_pos={0 0}, upload=411834]: Begin upload segment... Dec 25 04:40:39.297471084 (469h11m39.168554119s) [seg_pos={0 0}, upload=411834]: Done begin upload segment. Dec 25 04:40:39.297511581 (469h11m39.168594616s) [noise=true , address=145.79.227.51:4411 , piece_id=WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.297663310 (469h11m39.168746345s) [noise=true , address=145.79.227.17:4970 , piece_id=6DPKJ762IUN4A7UEZXLDERCVDW24QEKGEHFETAYZOKWQPVN2QJYA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.298405574 (469h11m39.169488619s) [noise=true , address=145.79.227.2:4114 , piece_id=5ISXZKQSH4CLLTF3ILGAXHV6STMLRUSZP6UATU2677X2FUPLBCRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.298797961 (469h11m39.169881006s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299124392 (469h11m39.170207437s) [noise=true , address=45.91.116.5:4300 , piece_id=NTWC3RWYQQBBKPMXGUG44R2B5GWC2Z4LDFU3GTWOWCT3JIRIZOOQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299438650 (469h11m39.170521685s) [noise=true , address=145.79.227.5:4221 , piece_id=C4SXUVTG2TZ4EIRBGL4TLOLBMYOAZ4KLIQSZOXKEOWAJBQQJLWSA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299794709 (469h11m39.170877744s) [noise=true , address=145.79.227.37:4567 , piece_id=TZ4C57ISUI3CB3VOLTSM5W5QVDI7PVXV67PFJI3ZSVSXHTKPYPSA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.300142221 (469h11m39.171225256s) [noise=true , address=145.79.227.8:4331 , piece_id=ELLVNJFOVY5HSAHZDQYGQVO7OYHFZNFHCBJKHQFV5K6EV2M3JTRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.300506354 (469h11m39.171589389s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.301142795 (469h11m39.17222584s) [noise=true , address=145.79.227.36:4517 , piece_id=HBQU7CMAO2KJS6ODW2WHICBAWQSH6TQQPOQSENKDEISWCWFN2CMA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.301625835 (469h11m39.17270891s) [noise=true , address=145.79.227.45:4229 , piece_id=GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.302170964 (469h11m39.173254009s) [noise=true , address=145.79.227.3:4150 , piece_id=OHGA4EA33AVEDCJLB6QSUJLD3W2TV6YNN6FBVFFKTTSF2B6XDPRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.302629947 (469h11m39.173712982s) [noise=true , address=145.79.227.19:4079 , piece_id=ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303079704 (469h11m39.174162749s) [noise=true , address=145.79.227.15:4875 , piece_id=FUMLT2ODHUE3ETSNWVKBYLSJXYVCHUW5KAJJE5AECNL3IAQYYCLQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303499804 (469h11m39.174582849s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303921386 (469h11m39.175004432s) [noise=true , address=145.79.227.47:4265 , piece_id=TVUXNR6K35A3CWFDN4JRKM4Q6RGBC4LP2XKPXJFN2AJL32ESEBPA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:41.172453460 (469h11m41.043536495s) [upload=411834]: Next returned nil segment Dec 25 04:40:41.172473798 (469h11m41.043556833s) [upload=411834]: Waiting for error group managing segments... Dec 25 04:40:41.483361556 (469h11m41.354444601s) [noise=true , address=145.79.227.37:4567 , piece_id=TZ4C57ISUI3CB3VOLTSM5W5QVDI7PVXV67PFJI3ZSVSXHTKPYPSA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.484283170 (469h11m41.355366205s) [noise=true , address=145.79.227.5:4221 , piece_id=C4SXUVTG2TZ4EIRBGL4TLOLBMYOAZ4KLIQSZOXKEOWAJBQQJLWSA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.515393699 (469h11m41.386476744s) [noise=true , address=145.79.227.51:4411 , piece_id=WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.526179387 (469h11m41.397262422s) [noise=true , address=45.91.116.5:4300 , piece_id=NTWC3RWYQQBBKPMXGUG44R2B5GWC2Z4LDFU3GTWOWCT3JIRIZOOQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.642380892 (469h11m41.513463937s) [noise=true , address=145.79.227.36:4517 , piece_id=HBQU7CMAO2KJS6ODW2WHICBAWQSH6TQQPOQSENKDEISWCWFN2CMA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.662326662 (469h11m41.533409707s) [noise=true , address=145.79.227.3:4150 , piece_id=OHGA4EA33AVEDCJLB6QSUJLD3W2TV6YNN6FBVFFKTTSF2B6XDPRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.758852130 (469h11m41.629935175s) [noise=true , address=145.79.227.47:4265 , piece_id=TVUXNR6K35A3CWFDN4JRKM4Q6RGBC4LP2XKPXJFN2AJL32ESEBPA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.823449082 (469h11m41.694532117s) [noise=true , address=145.79.227.17:4970 , piece_id=6DPKJ762IUN4A7UEZXLDERCVDW24QEKGEHFETAYZOKWQPVN2QJYA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.952636616 (469h11m41.823719661s) [noise=true , address=145.79.227.19:4079 , piece_id=ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.025177123 (469h11m41.896260168s) [noise=true , address=145.79.227.45:4229 , piece_id=GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.222526274 (469h11m42.093609319s) [noise=true , address=145.79.227.8:4331 , piece_id=ELLVNJFOVY5HSAHZDQYGQVO7OYHFZNFHCBJKHQFV5K6EV2M3JTRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.250024374 (469h11m42.121107409s) [noise=true , address=145.79.227.2:4114 , piece_id=5ISXZKQSH4CLLTF3ILGAXHV6STMLRUSZP6UATU2677X2FUPLBCRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.651787921 (469h11m42.522870966s) [noise=true , address=145.79.227.15:4875 , piece_id=FUMLT2ODHUE3ETSNWVKBYLSJXYVCHUW5KAJJE5AECNL3IAQYYCLQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.651814982 (469h11m42.522898027s) [seg_pos={0 0}, upload=411834]: Segment reached optimal threshold of 13 pieces. Dec 25 04:40:42.652052093 (469h11m42.523135129s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: write tcp 188.166.225.170:36650->77.90.31.15:4697: use of closed network connection; storage node overloaded, request limit: 2000 Dec 25 04:40:42.652069537 (469h11m42.523152572s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652177673 (469h11m42.523260708s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd): protocol: write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection; write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection Dec 25 04:40:42.652184456 (469h11m42.523267501s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652230243 (469h11m42.523313308s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq): protocol: write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection Dec 25 04:40:42.652240693 (469h11m42.523323778s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652268265 (469h11m42.52335131s) [seg_pos={0 0}, upload=411834]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 04:40:42.652354770 (469h11m42.523437815s) [upload=411834]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 04:40:42.652361693 (469h11m42.523444738s) [upload=411834]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 04:40:42.959704440 (469h11m42.830787485s) [upload=411834]: Done uploading segments... Dec 25 04:40:42.959728234 (469h11m42.83081127s) [upload=411834]: Upload finished. err: <nil> Duration: 5.941471149s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......................................................X....] duration: 3.481727338s trace: https://jaeger.dev.storj.io/trace/835f8cfb18d0547 FAIL (2025-12-25 05:17:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.481754241s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........................................................X...] duration: 3.551536277s trace: https://jaeger.dev.storj.io/trace/835f8cfb18d0547 FAIL (2025-12-25 05:17:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.551562628s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 5.932129349s trace: https://jaeger.dev.storj.io/trace/681cf842d2260d41 FAIL (2025-12-25 05:16:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, Piece ID: ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ) context deadline exceeded; piecestore: (Node ID: 1DT68m6J4PpDk3XUDhDaaqV46NLKVz4A23716JLFxEAh122GeR, Piece ID: WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA) context deadline exceeded; piecestore: (Node ID: 12wWYnhWHSqqAFgje4p9Cr7WYaJ6dJWsCqvrx8ZqVFoakjuFeka, Piece ID: GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q) context deadline exceeded ======LOGS====== nline: false Dec 25 04:40:39.039085560 (469h11m38.910168605s) [upload=411834]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 04:40:39.039095028 (469h11m38.910178073s) [upload=411834]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 04:40:39.297407613 (469h11m39.168490658s) [seg_pos={0 0}, upload=411834]: Begin upload segment... Dec 25 04:40:39.297471084 (469h11m39.168554119s) [seg_pos={0 0}, upload=411834]: Done begin upload segment. Dec 25 04:40:39.297511581 (469h11m39.168594616s) [noise=true , address=145.79.227.51:4411 , piece_id=WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.297663310 (469h11m39.168746345s) [noise=true , address=145.79.227.17:4970 , piece_id=6DPKJ762IUN4A7UEZXLDERCVDW24QEKGEHFETAYZOKWQPVN2QJYA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.298405574 (469h11m39.169488619s) [noise=true , address=145.79.227.2:4114 , piece_id=5ISXZKQSH4CLLTF3ILGAXHV6STMLRUSZP6UATU2677X2FUPLBCRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.298797961 (469h11m39.169881006s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299124392 (469h11m39.170207437s) [noise=true , address=45.91.116.5:4300 , piece_id=NTWC3RWYQQBBKPMXGUG44R2B5GWC2Z4LDFU3GTWOWCT3JIRIZOOQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299438650 (469h11m39.170521685s) [noise=true , address=145.79.227.5:4221 , piece_id=C4SXUVTG2TZ4EIRBGL4TLOLBMYOAZ4KLIQSZOXKEOWAJBQQJLWSA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299794709 (469h11m39.170877744s) [noise=true , address=145.79.227.37:4567 , piece_id=TZ4C57ISUI3CB3VOLTSM5W5QVDI7PVXV67PFJI3ZSVSXHTKPYPSA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.300142221 (469h11m39.171225256s) [noise=true , address=145.79.227.8:4331 , piece_id=ELLVNJFOVY5HSAHZDQYGQVO7OYHFZNFHCBJKHQFV5K6EV2M3JTRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.300506354 (469h11m39.171589389s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.301142795 (469h11m39.17222584s) [noise=true , address=145.79.227.36:4517 , piece_id=HBQU7CMAO2KJS6ODW2WHICBAWQSH6TQQPOQSENKDEISWCWFN2CMA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.301625835 (469h11m39.17270891s) [noise=true , address=145.79.227.45:4229 , piece_id=GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.302170964 (469h11m39.173254009s) [noise=true , address=145.79.227.3:4150 , piece_id=OHGA4EA33AVEDCJLB6QSUJLD3W2TV6YNN6FBVFFKTTSF2B6XDPRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.302629947 (469h11m39.173712982s) [noise=true , address=145.79.227.19:4079 , piece_id=ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303079704 (469h11m39.174162749s) [noise=true , address=145.79.227.15:4875 , piece_id=FUMLT2ODHUE3ETSNWVKBYLSJXYVCHUW5KAJJE5AECNL3IAQYYCLQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303499804 (469h11m39.174582849s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303921386 (469h11m39.175004432s) [noise=true , address=145.79.227.47:4265 , piece_id=TVUXNR6K35A3CWFDN4JRKM4Q6RGBC4LP2XKPXJFN2AJL32ESEBPA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:41.172453460 (469h11m41.043536495s) [upload=411834]: Next returned nil segment Dec 25 04:40:41.172473798 (469h11m41.043556833s) [upload=411834]: Waiting for error group managing segments... Dec 25 04:40:41.483361556 (469h11m41.354444601s) [noise=true , address=145.79.227.37:4567 , piece_id=TZ4C57ISUI3CB3VOLTSM5W5QVDI7PVXV67PFJI3ZSVSXHTKPYPSA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.484283170 (469h11m41.355366205s) [noise=true , address=145.79.227.5:4221 , piece_id=C4SXUVTG2TZ4EIRBGL4TLOLBMYOAZ4KLIQSZOXKEOWAJBQQJLWSA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.515393699 (469h11m41.386476744s) [noise=true , address=145.79.227.51:4411 , piece_id=WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.526179387 (469h11m41.397262422s) [noise=true , address=45.91.116.5:4300 , piece_id=NTWC3RWYQQBBKPMXGUG44R2B5GWC2Z4LDFU3GTWOWCT3JIRIZOOQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.642380892 (469h11m41.513463937s) [noise=true , address=145.79.227.36:4517 , piece_id=HBQU7CMAO2KJS6ODW2WHICBAWQSH6TQQPOQSENKDEISWCWFN2CMA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.662326662 (469h11m41.533409707s) [noise=true , address=145.79.227.3:4150 , piece_id=OHGA4EA33AVEDCJLB6QSUJLD3W2TV6YNN6FBVFFKTTSF2B6XDPRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.758852130 (469h11m41.629935175s) [noise=true , address=145.79.227.47:4265 , piece_id=TVUXNR6K35A3CWFDN4JRKM4Q6RGBC4LP2XKPXJFN2AJL32ESEBPA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.823449082 (469h11m41.694532117s) [noise=true , address=145.79.227.17:4970 , piece_id=6DPKJ762IUN4A7UEZXLDERCVDW24QEKGEHFETAYZOKWQPVN2QJYA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.952636616 (469h11m41.823719661s) [noise=true , address=145.79.227.19:4079 , piece_id=ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.025177123 (469h11m41.896260168s) [noise=true , address=145.79.227.45:4229 , piece_id=GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.222526274 (469h11m42.093609319s) [noise=true , address=145.79.227.8:4331 , piece_id=ELLVNJFOVY5HSAHZDQYGQVO7OYHFZNFHCBJKHQFV5K6EV2M3JTRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.250024374 (469h11m42.121107409s) [noise=true , address=145.79.227.2:4114 , piece_id=5ISXZKQSH4CLLTF3ILGAXHV6STMLRUSZP6UATU2677X2FUPLBCRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.651787921 (469h11m42.522870966s) [noise=true , address=145.79.227.15:4875 , piece_id=FUMLT2ODHUE3ETSNWVKBYLSJXYVCHUW5KAJJE5AECNL3IAQYYCLQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.651814982 (469h11m42.522898027s) [seg_pos={0 0}, upload=411834]: Segment reached optimal threshold of 13 pieces. Dec 25 04:40:42.652052093 (469h11m42.523135129s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: write tcp 188.166.225.170:36650->77.90.31.15:4697: use of closed network connection; storage node overloaded, request limit: 2000 Dec 25 04:40:42.652069537 (469h11m42.523152572s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652177673 (469h11m42.523260708s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd): protocol: write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection; write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection Dec 25 04:40:42.652184456 (469h11m42.523267501s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652230243 (469h11m42.523313308s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq): protocol: write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection Dec 25 04:40:42.652240693 (469h11m42.523323778s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652268265 (469h11m42.52335131s) [seg_pos={0 0}, upload=411834]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 04:40:42.652354770 (469h11m42.523437815s) [upload=411834]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 04:40:42.652361693 (469h11m42.523444738s) [upload=411834]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 04:40:42.959704440 (469h11m42.830787485s) [upload=411834]: Done uploading segments... Dec 25 04:40:42.959728234 (469h11m42.83081127s) [upload=411834]: Upload finished. err: <nil> Duration: 5.932159697s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........................................................X..] duration: 3.389985324s trace: https://jaeger.dev.storj.io/trace/835f8cfb18d0547 FAIL (2025-12-25 05:16:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.39001905s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 5.951991401s trace: https://jaeger.dev.storj.io/trace/681cf842d2260d41 FAIL (2025-12-25 05:16:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, Piece ID: ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ) context deadline exceeded; piecestore: (Node ID: 1DT68m6J4PpDk3XUDhDaaqV46NLKVz4A23716JLFxEAh122GeR, Piece ID: WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA) context deadline exceeded; piecestore: (Node ID: 12wWYnhWHSqqAFgje4p9Cr7WYaJ6dJWsCqvrx8ZqVFoakjuFeka, Piece ID: GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q) context deadline exceeded ======LOGS====== nline: false Dec 25 04:40:39.039085560 (469h11m38.910168605s) [upload=411834]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 04:40:39.039095028 (469h11m38.910178073s) [upload=411834]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 04:40:39.297407613 (469h11m39.168490658s) [seg_pos={0 0}, upload=411834]: Begin upload segment... Dec 25 04:40:39.297471084 (469h11m39.168554119s) [seg_pos={0 0}, upload=411834]: Done begin upload segment. Dec 25 04:40:39.297511581 (469h11m39.168594616s) [noise=true , address=145.79.227.51:4411 , piece_id=WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.297663310 (469h11m39.168746345s) [noise=true , address=145.79.227.17:4970 , piece_id=6DPKJ762IUN4A7UEZXLDERCVDW24QEKGEHFETAYZOKWQPVN2QJYA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.298405574 (469h11m39.169488619s) [noise=true , address=145.79.227.2:4114 , piece_id=5ISXZKQSH4CLLTF3ILGAXHV6STMLRUSZP6UATU2677X2FUPLBCRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.298797961 (469h11m39.169881006s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299124392 (469h11m39.170207437s) [noise=true , address=45.91.116.5:4300 , piece_id=NTWC3RWYQQBBKPMXGUG44R2B5GWC2Z4LDFU3GTWOWCT3JIRIZOOQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299438650 (469h11m39.170521685s) [noise=true , address=145.79.227.5:4221 , piece_id=C4SXUVTG2TZ4EIRBGL4TLOLBMYOAZ4KLIQSZOXKEOWAJBQQJLWSA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299794709 (469h11m39.170877744s) [noise=true , address=145.79.227.37:4567 , piece_id=TZ4C57ISUI3CB3VOLTSM5W5QVDI7PVXV67PFJI3ZSVSXHTKPYPSA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.300142221 (469h11m39.171225256s) [noise=true , address=145.79.227.8:4331 , piece_id=ELLVNJFOVY5HSAHZDQYGQVO7OYHFZNFHCBJKHQFV5K6EV2M3JTRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.300506354 (469h11m39.171589389s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.301142795 (469h11m39.17222584s) [noise=true , address=145.79.227.36:4517 , piece_id=HBQU7CMAO2KJS6ODW2WHICBAWQSH6TQQPOQSENKDEISWCWFN2CMA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.301625835 (469h11m39.17270891s) [noise=true , address=145.79.227.45:4229 , piece_id=GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.302170964 (469h11m39.173254009s) [noise=true , address=145.79.227.3:4150 , piece_id=OHGA4EA33AVEDCJLB6QSUJLD3W2TV6YNN6FBVFFKTTSF2B6XDPRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.302629947 (469h11m39.173712982s) [noise=true , address=145.79.227.19:4079 , piece_id=ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303079704 (469h11m39.174162749s) [noise=true , address=145.79.227.15:4875 , piece_id=FUMLT2ODHUE3ETSNWVKBYLSJXYVCHUW5KAJJE5AECNL3IAQYYCLQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303499804 (469h11m39.174582849s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303921386 (469h11m39.175004432s) [noise=true , address=145.79.227.47:4265 , piece_id=TVUXNR6K35A3CWFDN4JRKM4Q6RGBC4LP2XKPXJFN2AJL32ESEBPA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:41.172453460 (469h11m41.043536495s) [upload=411834]: Next returned nil segment Dec 25 04:40:41.172473798 (469h11m41.043556833s) [upload=411834]: Waiting for error group managing segments... Dec 25 04:40:41.483361556 (469h11m41.354444601s) [noise=true , address=145.79.227.37:4567 , piece_id=TZ4C57ISUI3CB3VOLTSM5W5QVDI7PVXV67PFJI3ZSVSXHTKPYPSA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.484283170 (469h11m41.355366205s) [noise=true , address=145.79.227.5:4221 , piece_id=C4SXUVTG2TZ4EIRBGL4TLOLBMYOAZ4KLIQSZOXKEOWAJBQQJLWSA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.515393699 (469h11m41.386476744s) [noise=true , address=145.79.227.51:4411 , piece_id=WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.526179387 (469h11m41.397262422s) [noise=true , address=45.91.116.5:4300 , piece_id=NTWC3RWYQQBBKPMXGUG44R2B5GWC2Z4LDFU3GTWOWCT3JIRIZOOQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.642380892 (469h11m41.513463937s) [noise=true , address=145.79.227.36:4517 , piece_id=HBQU7CMAO2KJS6ODW2WHICBAWQSH6TQQPOQSENKDEISWCWFN2CMA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.662326662 (469h11m41.533409707s) [noise=true , address=145.79.227.3:4150 , piece_id=OHGA4EA33AVEDCJLB6QSUJLD3W2TV6YNN6FBVFFKTTSF2B6XDPRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.758852130 (469h11m41.629935175s) [noise=true , address=145.79.227.47:4265 , piece_id=TVUXNR6K35A3CWFDN4JRKM4Q6RGBC4LP2XKPXJFN2AJL32ESEBPA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.823449082 (469h11m41.694532117s) [noise=true , address=145.79.227.17:4970 , piece_id=6DPKJ762IUN4A7UEZXLDERCVDW24QEKGEHFETAYZOKWQPVN2QJYA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.952636616 (469h11m41.823719661s) [noise=true , address=145.79.227.19:4079 , piece_id=ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.025177123 (469h11m41.896260168s) [noise=true , address=145.79.227.45:4229 , piece_id=GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.222526274 (469h11m42.093609319s) [noise=true , address=145.79.227.8:4331 , piece_id=ELLVNJFOVY5HSAHZDQYGQVO7OYHFZNFHCBJKHQFV5K6EV2M3JTRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.250024374 (469h11m42.121107409s) [noise=true , address=145.79.227.2:4114 , piece_id=5ISXZKQSH4CLLTF3ILGAXHV6STMLRUSZP6UATU2677X2FUPLBCRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.651787921 (469h11m42.522870966s) [noise=true , address=145.79.227.15:4875 , piece_id=FUMLT2ODHUE3ETSNWVKBYLSJXYVCHUW5KAJJE5AECNL3IAQYYCLQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.651814982 (469h11m42.522898027s) [seg_pos={0 0}, upload=411834]: Segment reached optimal threshold of 13 pieces. Dec 25 04:40:42.652052093 (469h11m42.523135129s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: write tcp 188.166.225.170:36650->77.90.31.15:4697: use of closed network connection; storage node overloaded, request limit: 2000 Dec 25 04:40:42.652069537 (469h11m42.523152572s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652177673 (469h11m42.523260708s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd): protocol: write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection; write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection Dec 25 04:40:42.652184456 (469h11m42.523267501s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652230243 (469h11m42.523313308s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq): protocol: write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection Dec 25 04:40:42.652240693 (469h11m42.523323778s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652268265 (469h11m42.52335131s) [seg_pos={0 0}, upload=411834]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 04:40:42.652354770 (469h11m42.523437815s) [upload=411834]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 04:40:42.652361693 (469h11m42.523444738s) [upload=411834]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 04:40:42.959704440 (469h11m42.830787485s) [upload=411834]: Done uploading segments... Dec 25 04:40:42.959728234 (469h11m42.83081127s) [upload=411834]: Upload finished. err: <nil> Duration: 5.952020846s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........................................................X.] duration: 3.536596715s trace: https://jaeger.dev.storj.io/trace/835f8cfb18d0547 FAIL (2025-12-25 05:15:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.536666049s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 5.8641831s trace: https://jaeger.dev.storj.io/trace/681cf842d2260d41 FAIL (2025-12-25 05:15:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, Piece ID: ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ) context deadline exceeded; piecestore: (Node ID: 1DT68m6J4PpDk3XUDhDaaqV46NLKVz4A23716JLFxEAh122GeR, Piece ID: WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA) context deadline exceeded; piecestore: (Node ID: 12wWYnhWHSqqAFgje4p9Cr7WYaJ6dJWsCqvrx8ZqVFoakjuFeka, Piece ID: GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q) context deadline exceeded ======LOGS====== nline: false Dec 25 04:40:39.039085560 (469h11m38.910168605s) [upload=411834]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 04:40:39.039095028 (469h11m38.910178073s) [upload=411834]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 04:40:39.297407613 (469h11m39.168490658s) [seg_pos={0 0}, upload=411834]: Begin upload segment... Dec 25 04:40:39.297471084 (469h11m39.168554119s) [seg_pos={0 0}, upload=411834]: Done begin upload segment. Dec 25 04:40:39.297511581 (469h11m39.168594616s) [noise=true , address=145.79.227.51:4411 , piece_id=WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.297663310 (469h11m39.168746345s) [noise=true , address=145.79.227.17:4970 , piece_id=6DPKJ762IUN4A7UEZXLDERCVDW24QEKGEHFETAYZOKWQPVN2QJYA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.298405574 (469h11m39.169488619s) [noise=true , address=145.79.227.2:4114 , piece_id=5ISXZKQSH4CLLTF3ILGAXHV6STMLRUSZP6UATU2677X2FUPLBCRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.298797961 (469h11m39.169881006s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299124392 (469h11m39.170207437s) [noise=true , address=45.91.116.5:4300 , piece_id=NTWC3RWYQQBBKPMXGUG44R2B5GWC2Z4LDFU3GTWOWCT3JIRIZOOQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299438650 (469h11m39.170521685s) [noise=true , address=145.79.227.5:4221 , piece_id=C4SXUVTG2TZ4EIRBGL4TLOLBMYOAZ4KLIQSZOXKEOWAJBQQJLWSA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299794709 (469h11m39.170877744s) [noise=true , address=145.79.227.37:4567 , piece_id=TZ4C57ISUI3CB3VOLTSM5W5QVDI7PVXV67PFJI3ZSVSXHTKPYPSA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.300142221 (469h11m39.171225256s) [noise=true , address=145.79.227.8:4331 , piece_id=ELLVNJFOVY5HSAHZDQYGQVO7OYHFZNFHCBJKHQFV5K6EV2M3JTRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.300506354 (469h11m39.171589389s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.301142795 (469h11m39.17222584s) [noise=true , address=145.79.227.36:4517 , piece_id=HBQU7CMAO2KJS6ODW2WHICBAWQSH6TQQPOQSENKDEISWCWFN2CMA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.301625835 (469h11m39.17270891s) [noise=true , address=145.79.227.45:4229 , piece_id=GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.302170964 (469h11m39.173254009s) [noise=true , address=145.79.227.3:4150 , piece_id=OHGA4EA33AVEDCJLB6QSUJLD3W2TV6YNN6FBVFFKTTSF2B6XDPRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.302629947 (469h11m39.173712982s) [noise=true , address=145.79.227.19:4079 , piece_id=ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303079704 (469h11m39.174162749s) [noise=true , address=145.79.227.15:4875 , piece_id=FUMLT2ODHUE3ETSNWVKBYLSJXYVCHUW5KAJJE5AECNL3IAQYYCLQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303499804 (469h11m39.174582849s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303921386 (469h11m39.175004432s) [noise=true , address=145.79.227.47:4265 , piece_id=TVUXNR6K35A3CWFDN4JRKM4Q6RGBC4LP2XKPXJFN2AJL32ESEBPA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:41.172453460 (469h11m41.043536495s) [upload=411834]: Next returned nil segment Dec 25 04:40:41.172473798 (469h11m41.043556833s) [upload=411834]: Waiting for error group managing segments... Dec 25 04:40:41.483361556 (469h11m41.354444601s) [noise=true , address=145.79.227.37:4567 , piece_id=TZ4C57ISUI3CB3VOLTSM5W5QVDI7PVXV67PFJI3ZSVSXHTKPYPSA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.484283170 (469h11m41.355366205s) [noise=true , address=145.79.227.5:4221 , piece_id=C4SXUVTG2TZ4EIRBGL4TLOLBMYOAZ4KLIQSZOXKEOWAJBQQJLWSA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.515393699 (469h11m41.386476744s) [noise=true , address=145.79.227.51:4411 , piece_id=WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.526179387 (469h11m41.397262422s) [noise=true , address=45.91.116.5:4300 , piece_id=NTWC3RWYQQBBKPMXGUG44R2B5GWC2Z4LDFU3GTWOWCT3JIRIZOOQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.642380892 (469h11m41.513463937s) [noise=true , address=145.79.227.36:4517 , piece_id=HBQU7CMAO2KJS6ODW2WHICBAWQSH6TQQPOQSENKDEISWCWFN2CMA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.662326662 (469h11m41.533409707s) [noise=true , address=145.79.227.3:4150 , piece_id=OHGA4EA33AVEDCJLB6QSUJLD3W2TV6YNN6FBVFFKTTSF2B6XDPRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.758852130 (469h11m41.629935175s) [noise=true , address=145.79.227.47:4265 , piece_id=TVUXNR6K35A3CWFDN4JRKM4Q6RGBC4LP2XKPXJFN2AJL32ESEBPA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.823449082 (469h11m41.694532117s) [noise=true , address=145.79.227.17:4970 , piece_id=6DPKJ762IUN4A7UEZXLDERCVDW24QEKGEHFETAYZOKWQPVN2QJYA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.952636616 (469h11m41.823719661s) [noise=true , address=145.79.227.19:4079 , piece_id=ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.025177123 (469h11m41.896260168s) [noise=true , address=145.79.227.45:4229 , piece_id=GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.222526274 (469h11m42.093609319s) [noise=true , address=145.79.227.8:4331 , piece_id=ELLVNJFOVY5HSAHZDQYGQVO7OYHFZNFHCBJKHQFV5K6EV2M3JTRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.250024374 (469h11m42.121107409s) [noise=true , address=145.79.227.2:4114 , piece_id=5ISXZKQSH4CLLTF3ILGAXHV6STMLRUSZP6UATU2677X2FUPLBCRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.651787921 (469h11m42.522870966s) [noise=true , address=145.79.227.15:4875 , piece_id=FUMLT2ODHUE3ETSNWVKBYLSJXYVCHUW5KAJJE5AECNL3IAQYYCLQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.651814982 (469h11m42.522898027s) [seg_pos={0 0}, upload=411834]: Segment reached optimal threshold of 13 pieces. Dec 25 04:40:42.652052093 (469h11m42.523135129s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: write tcp 188.166.225.170:36650->77.90.31.15:4697: use of closed network connection; storage node overloaded, request limit: 2000 Dec 25 04:40:42.652069537 (469h11m42.523152572s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652177673 (469h11m42.523260708s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd): protocol: write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection; write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection Dec 25 04:40:42.652184456 (469h11m42.523267501s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652230243 (469h11m42.523313308s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq): protocol: write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection Dec 25 04:40:42.652240693 (469h11m42.523323778s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652268265 (469h11m42.52335131s) [seg_pos={0 0}, upload=411834]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 04:40:42.652354770 (469h11m42.523437815s) [upload=411834]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 04:40:42.652361693 (469h11m42.523444738s) [upload=411834]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 04:40:42.959704440 (469h11m42.830787485s) [upload=411834]: Done uploading segments... Dec 25 04:40:42.959728234 (469h11m42.83081127s) [upload=411834]: Upload finished. err: <nil> Duration: 5.864210883s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 5.864166868s trace: https://jaeger.dev.storj.io/trace/681cf842d2260d41 FAIL (2025-12-25 05:15:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, Piece ID: ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ) context deadline exceeded; piecestore: (Node ID: 1DT68m6J4PpDk3XUDhDaaqV46NLKVz4A23716JLFxEAh122GeR, Piece ID: WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA) context deadline exceeded; piecestore: (Node ID: 12wWYnhWHSqqAFgje4p9Cr7WYaJ6dJWsCqvrx8ZqVFoakjuFeka, Piece ID: GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q) context deadline exceeded ======LOGS====== nline: false Dec 25 04:40:39.039085560 (469h11m38.910168605s) [upload=411834]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 04:40:39.039095028 (469h11m38.910178073s) [upload=411834]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 04:40:39.297407613 (469h11m39.168490658s) [seg_pos={0 0}, upload=411834]: Begin upload segment... Dec 25 04:40:39.297471084 (469h11m39.168554119s) [seg_pos={0 0}, upload=411834]: Done begin upload segment. Dec 25 04:40:39.297511581 (469h11m39.168594616s) [noise=true , address=145.79.227.51:4411 , piece_id=WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.297663310 (469h11m39.168746345s) [noise=true , address=145.79.227.17:4970 , piece_id=6DPKJ762IUN4A7UEZXLDERCVDW24QEKGEHFETAYZOKWQPVN2QJYA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.298405574 (469h11m39.169488619s) [noise=true , address=145.79.227.2:4114 , piece_id=5ISXZKQSH4CLLTF3ILGAXHV6STMLRUSZP6UATU2677X2FUPLBCRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.298797961 (469h11m39.169881006s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299124392 (469h11m39.170207437s) [noise=true , address=45.91.116.5:4300 , piece_id=NTWC3RWYQQBBKPMXGUG44R2B5GWC2Z4LDFU3GTWOWCT3JIRIZOOQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299438650 (469h11m39.170521685s) [noise=true , address=145.79.227.5:4221 , piece_id=C4SXUVTG2TZ4EIRBGL4TLOLBMYOAZ4KLIQSZOXKEOWAJBQQJLWSA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299794709 (469h11m39.170877744s) [noise=true , address=145.79.227.37:4567 , piece_id=TZ4C57ISUI3CB3VOLTSM5W5QVDI7PVXV67PFJI3ZSVSXHTKPYPSA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.300142221 (469h11m39.171225256s) [noise=true , address=145.79.227.8:4331 , piece_id=ELLVNJFOVY5HSAHZDQYGQVO7OYHFZNFHCBJKHQFV5K6EV2M3JTRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.300506354 (469h11m39.171589389s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.301142795 (469h11m39.17222584s) [noise=true , address=145.79.227.36:4517 , piece_id=HBQU7CMAO2KJS6ODW2WHICBAWQSH6TQQPOQSENKDEISWCWFN2CMA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.301625835 (469h11m39.17270891s) [noise=true , address=145.79.227.45:4229 , piece_id=GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.302170964 (469h11m39.173254009s) [noise=true , address=145.79.227.3:4150 , piece_id=OHGA4EA33AVEDCJLB6QSUJLD3W2TV6YNN6FBVFFKTTSF2B6XDPRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.302629947 (469h11m39.173712982s) [noise=true , address=145.79.227.19:4079 , piece_id=ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303079704 (469h11m39.174162749s) [noise=true , address=145.79.227.15:4875 , piece_id=FUMLT2ODHUE3ETSNWVKBYLSJXYVCHUW5KAJJE5AECNL3IAQYYCLQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303499804 (469h11m39.174582849s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303921386 (469h11m39.175004432s) [noise=true , address=145.79.227.47:4265 , piece_id=TVUXNR6K35A3CWFDN4JRKM4Q6RGBC4LP2XKPXJFN2AJL32ESEBPA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:41.172453460 (469h11m41.043536495s) [upload=411834]: Next returned nil segment Dec 25 04:40:41.172473798 (469h11m41.043556833s) [upload=411834]: Waiting for error group managing segments... Dec 25 04:40:41.483361556 (469h11m41.354444601s) [noise=true , address=145.79.227.37:4567 , piece_id=TZ4C57ISUI3CB3VOLTSM5W5QVDI7PVXV67PFJI3ZSVSXHTKPYPSA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.484283170 (469h11m41.355366205s) [noise=true , address=145.79.227.5:4221 , piece_id=C4SXUVTG2TZ4EIRBGL4TLOLBMYOAZ4KLIQSZOXKEOWAJBQQJLWSA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.515393699 (469h11m41.386476744s) [noise=true , address=145.79.227.51:4411 , piece_id=WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.526179387 (469h11m41.397262422s) [noise=true , address=45.91.116.5:4300 , piece_id=NTWC3RWYQQBBKPMXGUG44R2B5GWC2Z4LDFU3GTWOWCT3JIRIZOOQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.642380892 (469h11m41.513463937s) [noise=true , address=145.79.227.36:4517 , piece_id=HBQU7CMAO2KJS6ODW2WHICBAWQSH6TQQPOQSENKDEISWCWFN2CMA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.662326662 (469h11m41.533409707s) [noise=true , address=145.79.227.3:4150 , piece_id=OHGA4EA33AVEDCJLB6QSUJLD3W2TV6YNN6FBVFFKTTSF2B6XDPRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.758852130 (469h11m41.629935175s) [noise=true , address=145.79.227.47:4265 , piece_id=TVUXNR6K35A3CWFDN4JRKM4Q6RGBC4LP2XKPXJFN2AJL32ESEBPA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.823449082 (469h11m41.694532117s) [noise=true , address=145.79.227.17:4970 , piece_id=6DPKJ762IUN4A7UEZXLDERCVDW24QEKGEHFETAYZOKWQPVN2QJYA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.952636616 (469h11m41.823719661s) [noise=true , address=145.79.227.19:4079 , piece_id=ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.025177123 (469h11m41.896260168s) [noise=true , address=145.79.227.45:4229 , piece_id=GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.222526274 (469h11m42.093609319s) [noise=true , address=145.79.227.8:4331 , piece_id=ELLVNJFOVY5HSAHZDQYGQVO7OYHFZNFHCBJKHQFV5K6EV2M3JTRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.250024374 (469h11m42.121107409s) [noise=true , address=145.79.227.2:4114 , piece_id=5ISXZKQSH4CLLTF3ILGAXHV6STMLRUSZP6UATU2677X2FUPLBCRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.651787921 (469h11m42.522870966s) [noise=true , address=145.79.227.15:4875 , piece_id=FUMLT2ODHUE3ETSNWVKBYLSJXYVCHUW5KAJJE5AECNL3IAQYYCLQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.651814982 (469h11m42.522898027s) [seg_pos={0 0}, upload=411834]: Segment reached optimal threshold of 13 pieces. Dec 25 04:40:42.652052093 (469h11m42.523135129s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: write tcp 188.166.225.170:36650->77.90.31.15:4697: use of closed network connection; storage node overloaded, request limit: 2000 Dec 25 04:40:42.652069537 (469h11m42.523152572s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652177673 (469h11m42.523260708s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd): protocol: write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection; write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection Dec 25 04:40:42.652184456 (469h11m42.523267501s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652230243 (469h11m42.523313308s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq): protocol: write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection Dec 25 04:40:42.652240693 (469h11m42.523323778s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652268265 (469h11m42.52335131s) [seg_pos={0 0}, upload=411834]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 04:40:42.652354770 (469h11m42.523437815s) [upload=411834]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 04:40:42.652361693 (469h11m42.523444738s) [upload=411834]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 04:40:42.959704440 (469h11m42.830787485s) [upload=411834]: Done uploading segments... Dec 25 04:40:42.959728234 (469h11m42.83081127s) [upload=411834]: Upload finished. err: <nil> Duration: 5.86419921s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........................................................X] duration: 5.000994893s trace: https://jaeger.dev.storj.io/trace/835f8cfb18d0547 FAIL (2025-12-25 05:14:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00101401s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 6.359736389s trace: https://jaeger.dev.storj.io/trace/681cf842d2260d41 FAIL (2025-12-25 05:13:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, Piece ID: ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ) context deadline exceeded; piecestore: (Node ID: 1DT68m6J4PpDk3XUDhDaaqV46NLKVz4A23716JLFxEAh122GeR, Piece ID: WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA) context deadline exceeded; piecestore: (Node ID: 12wWYnhWHSqqAFgje4p9Cr7WYaJ6dJWsCqvrx8ZqVFoakjuFeka, Piece ID: GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q) context deadline exceeded ======LOGS====== nline: false Dec 25 04:40:39.039085560 (469h11m38.910168605s) [upload=411834]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 04:40:39.039095028 (469h11m38.910178073s) [upload=411834]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 04:40:39.297407613 (469h11m39.168490658s) [seg_pos={0 0}, upload=411834]: Begin upload segment... Dec 25 04:40:39.297471084 (469h11m39.168554119s) [seg_pos={0 0}, upload=411834]: Done begin upload segment. Dec 25 04:40:39.297511581 (469h11m39.168594616s) [noise=true , address=145.79.227.51:4411 , piece_id=WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.297663310 (469h11m39.168746345s) [noise=true , address=145.79.227.17:4970 , piece_id=6DPKJ762IUN4A7UEZXLDERCVDW24QEKGEHFETAYZOKWQPVN2QJYA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.298405574 (469h11m39.169488619s) [noise=true , address=145.79.227.2:4114 , piece_id=5ISXZKQSH4CLLTF3ILGAXHV6STMLRUSZP6UATU2677X2FUPLBCRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.298797961 (469h11m39.169881006s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299124392 (469h11m39.170207437s) [noise=true , address=45.91.116.5:4300 , piece_id=NTWC3RWYQQBBKPMXGUG44R2B5GWC2Z4LDFU3GTWOWCT3JIRIZOOQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299438650 (469h11m39.170521685s) [noise=true , address=145.79.227.5:4221 , piece_id=C4SXUVTG2TZ4EIRBGL4TLOLBMYOAZ4KLIQSZOXKEOWAJBQQJLWSA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299794709 (469h11m39.170877744s) [noise=true , address=145.79.227.37:4567 , piece_id=TZ4C57ISUI3CB3VOLTSM5W5QVDI7PVXV67PFJI3ZSVSXHTKPYPSA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.300142221 (469h11m39.171225256s) [noise=true , address=145.79.227.8:4331 , piece_id=ELLVNJFOVY5HSAHZDQYGQVO7OYHFZNFHCBJKHQFV5K6EV2M3JTRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.300506354 (469h11m39.171589389s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.301142795 (469h11m39.17222584s) [noise=true , address=145.79.227.36:4517 , piece_id=HBQU7CMAO2KJS6ODW2WHICBAWQSH6TQQPOQSENKDEISWCWFN2CMA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.301625835 (469h11m39.17270891s) [noise=true , address=145.79.227.45:4229 , piece_id=GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.302170964 (469h11m39.173254009s) [noise=true , address=145.79.227.3:4150 , piece_id=OHGA4EA33AVEDCJLB6QSUJLD3W2TV6YNN6FBVFFKTTSF2B6XDPRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.302629947 (469h11m39.173712982s) [noise=true , address=145.79.227.19:4079 , piece_id=ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303079704 (469h11m39.174162749s) [noise=true , address=145.79.227.15:4875 , piece_id=FUMLT2ODHUE3ETSNWVKBYLSJXYVCHUW5KAJJE5AECNL3IAQYYCLQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303499804 (469h11m39.174582849s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303921386 (469h11m39.175004432s) [noise=true , address=145.79.227.47:4265 , piece_id=TVUXNR6K35A3CWFDN4JRKM4Q6RGBC4LP2XKPXJFN2AJL32ESEBPA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:41.172453460 (469h11m41.043536495s) [upload=411834]: Next returned nil segment Dec 25 04:40:41.172473798 (469h11m41.043556833s) [upload=411834]: Waiting for error group managing segments... Dec 25 04:40:41.483361556 (469h11m41.354444601s) [noise=true , address=145.79.227.37:4567 , piece_id=TZ4C57ISUI3CB3VOLTSM5W5QVDI7PVXV67PFJI3ZSVSXHTKPYPSA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.484283170 (469h11m41.355366205s) [noise=true , address=145.79.227.5:4221 , piece_id=C4SXUVTG2TZ4EIRBGL4TLOLBMYOAZ4KLIQSZOXKEOWAJBQQJLWSA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.515393699 (469h11m41.386476744s) [noise=true , address=145.79.227.51:4411 , piece_id=WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.526179387 (469h11m41.397262422s) [noise=true , address=45.91.116.5:4300 , piece_id=NTWC3RWYQQBBKPMXGUG44R2B5GWC2Z4LDFU3GTWOWCT3JIRIZOOQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.642380892 (469h11m41.513463937s) [noise=true , address=145.79.227.36:4517 , piece_id=HBQU7CMAO2KJS6ODW2WHICBAWQSH6TQQPOQSENKDEISWCWFN2CMA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.662326662 (469h11m41.533409707s) [noise=true , address=145.79.227.3:4150 , piece_id=OHGA4EA33AVEDCJLB6QSUJLD3W2TV6YNN6FBVFFKTTSF2B6XDPRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.758852130 (469h11m41.629935175s) [noise=true , address=145.79.227.47:4265 , piece_id=TVUXNR6K35A3CWFDN4JRKM4Q6RGBC4LP2XKPXJFN2AJL32ESEBPA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.823449082 (469h11m41.694532117s) [noise=true , address=145.79.227.17:4970 , piece_id=6DPKJ762IUN4A7UEZXLDERCVDW24QEKGEHFETAYZOKWQPVN2QJYA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.952636616 (469h11m41.823719661s) [noise=true , address=145.79.227.19:4079 , piece_id=ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.025177123 (469h11m41.896260168s) [noise=true , address=145.79.227.45:4229 , piece_id=GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.222526274 (469h11m42.093609319s) [noise=true , address=145.79.227.8:4331 , piece_id=ELLVNJFOVY5HSAHZDQYGQVO7OYHFZNFHCBJKHQFV5K6EV2M3JTRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.250024374 (469h11m42.121107409s) [noise=true , address=145.79.227.2:4114 , piece_id=5ISXZKQSH4CLLTF3ILGAXHV6STMLRUSZP6UATU2677X2FUPLBCRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.651787921 (469h11m42.522870966s) [noise=true , address=145.79.227.15:4875 , piece_id=FUMLT2ODHUE3ETSNWVKBYLSJXYVCHUW5KAJJE5AECNL3IAQYYCLQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.651814982 (469h11m42.522898027s) [seg_pos={0 0}, upload=411834]: Segment reached optimal threshold of 13 pieces. Dec 25 04:40:42.652052093 (469h11m42.523135129s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: write tcp 188.166.225.170:36650->77.90.31.15:4697: use of closed network connection; storage node overloaded, request limit: 2000 Dec 25 04:40:42.652069537 (469h11m42.523152572s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652177673 (469h11m42.523260708s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd): protocol: write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection; write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection Dec 25 04:40:42.652184456 (469h11m42.523267501s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652230243 (469h11m42.523313308s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq): protocol: write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection Dec 25 04:40:42.652240693 (469h11m42.523323778s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652268265 (469h11m42.52335131s) [seg_pos={0 0}, upload=411834]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 04:40:42.652354770 (469h11m42.523437815s) [upload=411834]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 04:40:42.652361693 (469h11m42.523444738s) [upload=411834]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 04:40:42.959704440 (469h11m42.830787485s) [upload=411834]: Done uploading segments... Dec 25 04:40:42.959728234 (469h11m42.83081127s) [upload=411834]: Upload finished. err: <nil> Duration: 6.359763741s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 6.034509602s trace: https://jaeger.dev.storj.io/trace/681cf842d2260d41 FAIL (2025-12-25 05:13:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, Piece ID: ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ) context deadline exceeded; piecestore: (Node ID: 1DT68m6J4PpDk3XUDhDaaqV46NLKVz4A23716JLFxEAh122GeR, Piece ID: WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA) context deadline exceeded; piecestore: (Node ID: 12wWYnhWHSqqAFgje4p9Cr7WYaJ6dJWsCqvrx8ZqVFoakjuFeka, Piece ID: GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q) context deadline exceeded ======LOGS====== nline: false Dec 25 04:40:39.039085560 (469h11m38.910168605s) [upload=411834]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 04:40:39.039095028 (469h11m38.910178073s) [upload=411834]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 04:40:39.297407613 (469h11m39.168490658s) [seg_pos={0 0}, upload=411834]: Begin upload segment... Dec 25 04:40:39.297471084 (469h11m39.168554119s) [seg_pos={0 0}, upload=411834]: Done begin upload segment. Dec 25 04:40:39.297511581 (469h11m39.168594616s) [noise=true , address=145.79.227.51:4411 , piece_id=WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.297663310 (469h11m39.168746345s) [noise=true , address=145.79.227.17:4970 , piece_id=6DPKJ762IUN4A7UEZXLDERCVDW24QEKGEHFETAYZOKWQPVN2QJYA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.298405574 (469h11m39.169488619s) [noise=true , address=145.79.227.2:4114 , piece_id=5ISXZKQSH4CLLTF3ILGAXHV6STMLRUSZP6UATU2677X2FUPLBCRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.298797961 (469h11m39.169881006s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299124392 (469h11m39.170207437s) [noise=true , address=45.91.116.5:4300 , piece_id=NTWC3RWYQQBBKPMXGUG44R2B5GWC2Z4LDFU3GTWOWCT3JIRIZOOQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299438650 (469h11m39.170521685s) [noise=true , address=145.79.227.5:4221 , piece_id=C4SXUVTG2TZ4EIRBGL4TLOLBMYOAZ4KLIQSZOXKEOWAJBQQJLWSA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299794709 (469h11m39.170877744s) [noise=true , address=145.79.227.37:4567 , piece_id=TZ4C57ISUI3CB3VOLTSM5W5QVDI7PVXV67PFJI3ZSVSXHTKPYPSA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.300142221 (469h11m39.171225256s) [noise=true , address=145.79.227.8:4331 , piece_id=ELLVNJFOVY5HSAHZDQYGQVO7OYHFZNFHCBJKHQFV5K6EV2M3JTRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.300506354 (469h11m39.171589389s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.301142795 (469h11m39.17222584s) [noise=true , address=145.79.227.36:4517 , piece_id=HBQU7CMAO2KJS6ODW2WHICBAWQSH6TQQPOQSENKDEISWCWFN2CMA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.301625835 (469h11m39.17270891s) [noise=true , address=145.79.227.45:4229 , piece_id=GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.302170964 (469h11m39.173254009s) [noise=true , address=145.79.227.3:4150 , piece_id=OHGA4EA33AVEDCJLB6QSUJLD3W2TV6YNN6FBVFFKTTSF2B6XDPRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.302629947 (469h11m39.173712982s) [noise=true , address=145.79.227.19:4079 , piece_id=ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303079704 (469h11m39.174162749s) [noise=true , address=145.79.227.15:4875 , piece_id=FUMLT2ODHUE3ETSNWVKBYLSJXYVCHUW5KAJJE5AECNL3IAQYYCLQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303499804 (469h11m39.174582849s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303921386 (469h11m39.175004432s) [noise=true , address=145.79.227.47:4265 , piece_id=TVUXNR6K35A3CWFDN4JRKM4Q6RGBC4LP2XKPXJFN2AJL32ESEBPA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:41.172453460 (469h11m41.043536495s) [upload=411834]: Next returned nil segment Dec 25 04:40:41.172473798 (469h11m41.043556833s) [upload=411834]: Waiting for error group managing segments... Dec 25 04:40:41.483361556 (469h11m41.354444601s) [noise=true , address=145.79.227.37:4567 , piece_id=TZ4C57ISUI3CB3VOLTSM5W5QVDI7PVXV67PFJI3ZSVSXHTKPYPSA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.484283170 (469h11m41.355366205s) [noise=true , address=145.79.227.5:4221 , piece_id=C4SXUVTG2TZ4EIRBGL4TLOLBMYOAZ4KLIQSZOXKEOWAJBQQJLWSA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.515393699 (469h11m41.386476744s) [noise=true , address=145.79.227.51:4411 , piece_id=WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.526179387 (469h11m41.397262422s) [noise=true , address=45.91.116.5:4300 , piece_id=NTWC3RWYQQBBKPMXGUG44R2B5GWC2Z4LDFU3GTWOWCT3JIRIZOOQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.642380892 (469h11m41.513463937s) [noise=true , address=145.79.227.36:4517 , piece_id=HBQU7CMAO2KJS6ODW2WHICBAWQSH6TQQPOQSENKDEISWCWFN2CMA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.662326662 (469h11m41.533409707s) [noise=true , address=145.79.227.3:4150 , piece_id=OHGA4EA33AVEDCJLB6QSUJLD3W2TV6YNN6FBVFFKTTSF2B6XDPRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.758852130 (469h11m41.629935175s) [noise=true , address=145.79.227.47:4265 , piece_id=TVUXNR6K35A3CWFDN4JRKM4Q6RGBC4LP2XKPXJFN2AJL32ESEBPA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.823449082 (469h11m41.694532117s) [noise=true , address=145.79.227.17:4970 , piece_id=6DPKJ762IUN4A7UEZXLDERCVDW24QEKGEHFETAYZOKWQPVN2QJYA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.952636616 (469h11m41.823719661s) [noise=true , address=145.79.227.19:4079 , piece_id=ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.025177123 (469h11m41.896260168s) [noise=true , address=145.79.227.45:4229 , piece_id=GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.222526274 (469h11m42.093609319s) [noise=true , address=145.79.227.8:4331 , piece_id=ELLVNJFOVY5HSAHZDQYGQVO7OYHFZNFHCBJKHQFV5K6EV2M3JTRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.250024374 (469h11m42.121107409s) [noise=true , address=145.79.227.2:4114 , piece_id=5ISXZKQSH4CLLTF3ILGAXHV6STMLRUSZP6UATU2677X2FUPLBCRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.651787921 (469h11m42.522870966s) [noise=true , address=145.79.227.15:4875 , piece_id=FUMLT2ODHUE3ETSNWVKBYLSJXYVCHUW5KAJJE5AECNL3IAQYYCLQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.651814982 (469h11m42.522898027s) [seg_pos={0 0}, upload=411834]: Segment reached optimal threshold of 13 pieces. Dec 25 04:40:42.652052093 (469h11m42.523135129s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: write tcp 188.166.225.170:36650->77.90.31.15:4697: use of closed network connection; storage node overloaded, request limit: 2000 Dec 25 04:40:42.652069537 (469h11m42.523152572s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652177673 (469h11m42.523260708s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd): protocol: write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection; write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection Dec 25 04:40:42.652184456 (469h11m42.523267501s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652230243 (469h11m42.523313308s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq): protocol: write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection Dec 25 04:40:42.652240693 (469h11m42.523323778s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652268265 (469h11m42.52335131s) [seg_pos={0 0}, upload=411834]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 04:40:42.652354770 (469h11m42.523437815s) [upload=411834]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 04:40:42.652361693 (469h11m42.523444738s) [upload=411834]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 04:40:42.959704440 (469h11m42.830787485s) [upload=411834]: Done uploading segments... Dec 25 04:40:42.959728234 (469h11m42.83081127s) [upload=411834]: Upload finished. err: <nil> Duration: 6.034536091s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 6.092312714s trace: https://jaeger.dev.storj.io/trace/681cf842d2260d41 FAIL (2025-12-25 05:12:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, Piece ID: ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ) context deadline exceeded; piecestore: (Node ID: 1DT68m6J4PpDk3XUDhDaaqV46NLKVz4A23716JLFxEAh122GeR, Piece ID: WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA) context deadline exceeded; piecestore: (Node ID: 12wWYnhWHSqqAFgje4p9Cr7WYaJ6dJWsCqvrx8ZqVFoakjuFeka, Piece ID: GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q) context deadline exceeded ======LOGS====== nline: false Dec 25 04:40:39.039085560 (469h11m38.910168605s) [upload=411834]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 04:40:39.039095028 (469h11m38.910178073s) [upload=411834]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 04:40:39.297407613 (469h11m39.168490658s) [seg_pos={0 0}, upload=411834]: Begin upload segment... Dec 25 04:40:39.297471084 (469h11m39.168554119s) [seg_pos={0 0}, upload=411834]: Done begin upload segment. Dec 25 04:40:39.297511581 (469h11m39.168594616s) [noise=true , address=145.79.227.51:4411 , piece_id=WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.297663310 (469h11m39.168746345s) [noise=true , address=145.79.227.17:4970 , piece_id=6DPKJ762IUN4A7UEZXLDERCVDW24QEKGEHFETAYZOKWQPVN2QJYA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.298405574 (469h11m39.169488619s) [noise=true , address=145.79.227.2:4114 , piece_id=5ISXZKQSH4CLLTF3ILGAXHV6STMLRUSZP6UATU2677X2FUPLBCRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.298797961 (469h11m39.169881006s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299124392 (469h11m39.170207437s) [noise=true , address=45.91.116.5:4300 , piece_id=NTWC3RWYQQBBKPMXGUG44R2B5GWC2Z4LDFU3GTWOWCT3JIRIZOOQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299438650 (469h11m39.170521685s) [noise=true , address=145.79.227.5:4221 , piece_id=C4SXUVTG2TZ4EIRBGL4TLOLBMYOAZ4KLIQSZOXKEOWAJBQQJLWSA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299794709 (469h11m39.170877744s) [noise=true , address=145.79.227.37:4567 , piece_id=TZ4C57ISUI3CB3VOLTSM5W5QVDI7PVXV67PFJI3ZSVSXHTKPYPSA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.300142221 (469h11m39.171225256s) [noise=true , address=145.79.227.8:4331 , piece_id=ELLVNJFOVY5HSAHZDQYGQVO7OYHFZNFHCBJKHQFV5K6EV2M3JTRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.300506354 (469h11m39.171589389s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.301142795 (469h11m39.17222584s) [noise=true , address=145.79.227.36:4517 , piece_id=HBQU7CMAO2KJS6ODW2WHICBAWQSH6TQQPOQSENKDEISWCWFN2CMA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.301625835 (469h11m39.17270891s) [noise=true , address=145.79.227.45:4229 , piece_id=GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.302170964 (469h11m39.173254009s) [noise=true , address=145.79.227.3:4150 , piece_id=OHGA4EA33AVEDCJLB6QSUJLD3W2TV6YNN6FBVFFKTTSF2B6XDPRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.302629947 (469h11m39.173712982s) [noise=true , address=145.79.227.19:4079 , piece_id=ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303079704 (469h11m39.174162749s) [noise=true , address=145.79.227.15:4875 , piece_id=FUMLT2ODHUE3ETSNWVKBYLSJXYVCHUW5KAJJE5AECNL3IAQYYCLQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303499804 (469h11m39.174582849s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303921386 (469h11m39.175004432s) [noise=true , address=145.79.227.47:4265 , piece_id=TVUXNR6K35A3CWFDN4JRKM4Q6RGBC4LP2XKPXJFN2AJL32ESEBPA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:41.172453460 (469h11m41.043536495s) [upload=411834]: Next returned nil segment Dec 25 04:40:41.172473798 (469h11m41.043556833s) [upload=411834]: Waiting for error group managing segments... Dec 25 04:40:41.483361556 (469h11m41.354444601s) [noise=true , address=145.79.227.37:4567 , piece_id=TZ4C57ISUI3CB3VOLTSM5W5QVDI7PVXV67PFJI3ZSVSXHTKPYPSA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.484283170 (469h11m41.355366205s) [noise=true , address=145.79.227.5:4221 , piece_id=C4SXUVTG2TZ4EIRBGL4TLOLBMYOAZ4KLIQSZOXKEOWAJBQQJLWSA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.515393699 (469h11m41.386476744s) [noise=true , address=145.79.227.51:4411 , piece_id=WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.526179387 (469h11m41.397262422s) [noise=true , address=45.91.116.5:4300 , piece_id=NTWC3RWYQQBBKPMXGUG44R2B5GWC2Z4LDFU3GTWOWCT3JIRIZOOQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.642380892 (469h11m41.513463937s) [noise=true , address=145.79.227.36:4517 , piece_id=HBQU7CMAO2KJS6ODW2WHICBAWQSH6TQQPOQSENKDEISWCWFN2CMA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.662326662 (469h11m41.533409707s) [noise=true , address=145.79.227.3:4150 , piece_id=OHGA4EA33AVEDCJLB6QSUJLD3W2TV6YNN6FBVFFKTTSF2B6XDPRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.758852130 (469h11m41.629935175s) [noise=true , address=145.79.227.47:4265 , piece_id=TVUXNR6K35A3CWFDN4JRKM4Q6RGBC4LP2XKPXJFN2AJL32ESEBPA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.823449082 (469h11m41.694532117s) [noise=true , address=145.79.227.17:4970 , piece_id=6DPKJ762IUN4A7UEZXLDERCVDW24QEKGEHFETAYZOKWQPVN2QJYA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.952636616 (469h11m41.823719661s) [noise=true , address=145.79.227.19:4079 , piece_id=ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.025177123 (469h11m41.896260168s) [noise=true , address=145.79.227.45:4229 , piece_id=GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.222526274 (469h11m42.093609319s) [noise=true , address=145.79.227.8:4331 , piece_id=ELLVNJFOVY5HSAHZDQYGQVO7OYHFZNFHCBJKHQFV5K6EV2M3JTRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.250024374 (469h11m42.121107409s) [noise=true , address=145.79.227.2:4114 , piece_id=5ISXZKQSH4CLLTF3ILGAXHV6STMLRUSZP6UATU2677X2FUPLBCRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.651787921 (469h11m42.522870966s) [noise=true , address=145.79.227.15:4875 , piece_id=FUMLT2ODHUE3ETSNWVKBYLSJXYVCHUW5KAJJE5AECNL3IAQYYCLQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.651814982 (469h11m42.522898027s) [seg_pos={0 0}, upload=411834]: Segment reached optimal threshold of 13 pieces. Dec 25 04:40:42.652052093 (469h11m42.523135129s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: write tcp 188.166.225.170:36650->77.90.31.15:4697: use of closed network connection; storage node overloaded, request limit: 2000 Dec 25 04:40:42.652069537 (469h11m42.523152572s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652177673 (469h11m42.523260708s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd): protocol: write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection; write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection Dec 25 04:40:42.652184456 (469h11m42.523267501s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652230243 (469h11m42.523313308s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq): protocol: write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection Dec 25 04:40:42.652240693 (469h11m42.523323778s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652268265 (469h11m42.52335131s) [seg_pos={0 0}, upload=411834]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 04:40:42.652354770 (469h11m42.523437815s) [upload=411834]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 04:40:42.652361693 (469h11m42.523444738s) [upload=411834]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 04:40:42.959704440 (469h11m42.830787485s) [upload=411834]: Done uploading segments... Dec 25 04:40:42.959728234 (469h11m42.83081127s) [upload=411834]: Upload finished. err: <nil> Duration: 6.092335709s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 5.752410973s trace: https://jaeger.dev.storj.io/trace/681cf842d2260d41 FAIL (2025-12-25 05:11:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, Piece ID: ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ) context deadline exceeded; piecestore: (Node ID: 1DT68m6J4PpDk3XUDhDaaqV46NLKVz4A23716JLFxEAh122GeR, Piece ID: WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA) context deadline exceeded; piecestore: (Node ID: 12wWYnhWHSqqAFgje4p9Cr7WYaJ6dJWsCqvrx8ZqVFoakjuFeka, Piece ID: GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q) context deadline exceeded ======LOGS====== nline: false Dec 25 04:40:39.039085560 (469h11m38.910168605s) [upload=411834]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 04:40:39.039095028 (469h11m38.910178073s) [upload=411834]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 04:40:39.297407613 (469h11m39.168490658s) [seg_pos={0 0}, upload=411834]: Begin upload segment... Dec 25 04:40:39.297471084 (469h11m39.168554119s) [seg_pos={0 0}, upload=411834]: Done begin upload segment. Dec 25 04:40:39.297511581 (469h11m39.168594616s) [noise=true , address=145.79.227.51:4411 , piece_id=WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.297663310 (469h11m39.168746345s) [noise=true , address=145.79.227.17:4970 , piece_id=6DPKJ762IUN4A7UEZXLDERCVDW24QEKGEHFETAYZOKWQPVN2QJYA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.298405574 (469h11m39.169488619s) [noise=true , address=145.79.227.2:4114 , piece_id=5ISXZKQSH4CLLTF3ILGAXHV6STMLRUSZP6UATU2677X2FUPLBCRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.298797961 (469h11m39.169881006s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299124392 (469h11m39.170207437s) [noise=true , address=45.91.116.5:4300 , piece_id=NTWC3RWYQQBBKPMXGUG44R2B5GWC2Z4LDFU3GTWOWCT3JIRIZOOQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299438650 (469h11m39.170521685s) [noise=true , address=145.79.227.5:4221 , piece_id=C4SXUVTG2TZ4EIRBGL4TLOLBMYOAZ4KLIQSZOXKEOWAJBQQJLWSA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299794709 (469h11m39.170877744s) [noise=true , address=145.79.227.37:4567 , piece_id=TZ4C57ISUI3CB3VOLTSM5W5QVDI7PVXV67PFJI3ZSVSXHTKPYPSA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.300142221 (469h11m39.171225256s) [noise=true , address=145.79.227.8:4331 , piece_id=ELLVNJFOVY5HSAHZDQYGQVO7OYHFZNFHCBJKHQFV5K6EV2M3JTRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.300506354 (469h11m39.171589389s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.301142795 (469h11m39.17222584s) [noise=true , address=145.79.227.36:4517 , piece_id=HBQU7CMAO2KJS6ODW2WHICBAWQSH6TQQPOQSENKDEISWCWFN2CMA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.301625835 (469h11m39.17270891s) [noise=true , address=145.79.227.45:4229 , piece_id=GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.302170964 (469h11m39.173254009s) [noise=true , address=145.79.227.3:4150 , piece_id=OHGA4EA33AVEDCJLB6QSUJLD3W2TV6YNN6FBVFFKTTSF2B6XDPRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.302629947 (469h11m39.173712982s) [noise=true , address=145.79.227.19:4079 , piece_id=ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303079704 (469h11m39.174162749s) [noise=true , address=145.79.227.15:4875 , piece_id=FUMLT2ODHUE3ETSNWVKBYLSJXYVCHUW5KAJJE5AECNL3IAQYYCLQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303499804 (469h11m39.174582849s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303921386 (469h11m39.175004432s) [noise=true , address=145.79.227.47:4265 , piece_id=TVUXNR6K35A3CWFDN4JRKM4Q6RGBC4LP2XKPXJFN2AJL32ESEBPA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:41.172453460 (469h11m41.043536495s) [upload=411834]: Next returned nil segment Dec 25 04:40:41.172473798 (469h11m41.043556833s) [upload=411834]: Waiting for error group managing segments... Dec 25 04:40:41.483361556 (469h11m41.354444601s) [noise=true , address=145.79.227.37:4567 , piece_id=TZ4C57ISUI3CB3VOLTSM5W5QVDI7PVXV67PFJI3ZSVSXHTKPYPSA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.484283170 (469h11m41.355366205s) [noise=true , address=145.79.227.5:4221 , piece_id=C4SXUVTG2TZ4EIRBGL4TLOLBMYOAZ4KLIQSZOXKEOWAJBQQJLWSA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.515393699 (469h11m41.386476744s) [noise=true , address=145.79.227.51:4411 , piece_id=WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.526179387 (469h11m41.397262422s) [noise=true , address=45.91.116.5:4300 , piece_id=NTWC3RWYQQBBKPMXGUG44R2B5GWC2Z4LDFU3GTWOWCT3JIRIZOOQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.642380892 (469h11m41.513463937s) [noise=true , address=145.79.227.36:4517 , piece_id=HBQU7CMAO2KJS6ODW2WHICBAWQSH6TQQPOQSENKDEISWCWFN2CMA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.662326662 (469h11m41.533409707s) [noise=true , address=145.79.227.3:4150 , piece_id=OHGA4EA33AVEDCJLB6QSUJLD3W2TV6YNN6FBVFFKTTSF2B6XDPRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.758852130 (469h11m41.629935175s) [noise=true , address=145.79.227.47:4265 , piece_id=TVUXNR6K35A3CWFDN4JRKM4Q6RGBC4LP2XKPXJFN2AJL32ESEBPA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.823449082 (469h11m41.694532117s) [noise=true , address=145.79.227.17:4970 , piece_id=6DPKJ762IUN4A7UEZXLDERCVDW24QEKGEHFETAYZOKWQPVN2QJYA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.952636616 (469h11m41.823719661s) [noise=true , address=145.79.227.19:4079 , piece_id=ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.025177123 (469h11m41.896260168s) [noise=true , address=145.79.227.45:4229 , piece_id=GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.222526274 (469h11m42.093609319s) [noise=true , address=145.79.227.8:4331 , piece_id=ELLVNJFOVY5HSAHZDQYGQVO7OYHFZNFHCBJKHQFV5K6EV2M3JTRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.250024374 (469h11m42.121107409s) [noise=true , address=145.79.227.2:4114 , piece_id=5ISXZKQSH4CLLTF3ILGAXHV6STMLRUSZP6UATU2677X2FUPLBCRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.651787921 (469h11m42.522870966s) [noise=true , address=145.79.227.15:4875 , piece_id=FUMLT2ODHUE3ETSNWVKBYLSJXYVCHUW5KAJJE5AECNL3IAQYYCLQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.651814982 (469h11m42.522898027s) [seg_pos={0 0}, upload=411834]: Segment reached optimal threshold of 13 pieces. Dec 25 04:40:42.652052093 (469h11m42.523135129s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: write tcp 188.166.225.170:36650->77.90.31.15:4697: use of closed network connection; storage node overloaded, request limit: 2000 Dec 25 04:40:42.652069537 (469h11m42.523152572s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652177673 (469h11m42.523260708s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd): protocol: write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection; write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection Dec 25 04:40:42.652184456 (469h11m42.523267501s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652230243 (469h11m42.523313308s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq): protocol: write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection Dec 25 04:40:42.652240693 (469h11m42.523323778s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652268265 (469h11m42.52335131s) [seg_pos={0 0}, upload=411834]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 04:40:42.652354770 (469h11m42.523437815s) [upload=411834]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 04:40:42.652361693 (469h11m42.523444738s) [upload=411834]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 04:40:42.959704440 (469h11m42.830787485s) [upload=411834]: Done uploading segments... Dec 25 04:40:42.959728234 (469h11m42.83081127s) [upload=411834]: Upload finished. err: <nil> Duration: 5.752449056s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 5.919075085s trace: https://jaeger.dev.storj.io/trace/681cf842d2260d41 FAIL (2025-12-25 05:11:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, Piece ID: ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ) context deadline exceeded; piecestore: (Node ID: 1DT68m6J4PpDk3XUDhDaaqV46NLKVz4A23716JLFxEAh122GeR, Piece ID: WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA) context deadline exceeded; piecestore: (Node ID: 12wWYnhWHSqqAFgje4p9Cr7WYaJ6dJWsCqvrx8ZqVFoakjuFeka, Piece ID: GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q) context deadline exceeded ======LOGS====== nline: false Dec 25 04:40:39.039085560 (469h11m38.910168605s) [upload=411834]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 04:40:39.039095028 (469h11m38.910178073s) [upload=411834]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 04:40:39.297407613 (469h11m39.168490658s) [seg_pos={0 0}, upload=411834]: Begin upload segment... Dec 25 04:40:39.297471084 (469h11m39.168554119s) [seg_pos={0 0}, upload=411834]: Done begin upload segment. Dec 25 04:40:39.297511581 (469h11m39.168594616s) [noise=true , address=145.79.227.51:4411 , piece_id=WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.297663310 (469h11m39.168746345s) [noise=true , address=145.79.227.17:4970 , piece_id=6DPKJ762IUN4A7UEZXLDERCVDW24QEKGEHFETAYZOKWQPVN2QJYA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.298405574 (469h11m39.169488619s) [noise=true , address=145.79.227.2:4114 , piece_id=5ISXZKQSH4CLLTF3ILGAXHV6STMLRUSZP6UATU2677X2FUPLBCRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.298797961 (469h11m39.169881006s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299124392 (469h11m39.170207437s) [noise=true , address=45.91.116.5:4300 , piece_id=NTWC3RWYQQBBKPMXGUG44R2B5GWC2Z4LDFU3GTWOWCT3JIRIZOOQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299438650 (469h11m39.170521685s) [noise=true , address=145.79.227.5:4221 , piece_id=C4SXUVTG2TZ4EIRBGL4TLOLBMYOAZ4KLIQSZOXKEOWAJBQQJLWSA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299794709 (469h11m39.170877744s) [noise=true , address=145.79.227.37:4567 , piece_id=TZ4C57ISUI3CB3VOLTSM5W5QVDI7PVXV67PFJI3ZSVSXHTKPYPSA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.300142221 (469h11m39.171225256s) [noise=true , address=145.79.227.8:4331 , piece_id=ELLVNJFOVY5HSAHZDQYGQVO7OYHFZNFHCBJKHQFV5K6EV2M3JTRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.300506354 (469h11m39.171589389s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.301142795 (469h11m39.17222584s) [noise=true , address=145.79.227.36:4517 , piece_id=HBQU7CMAO2KJS6ODW2WHICBAWQSH6TQQPOQSENKDEISWCWFN2CMA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.301625835 (469h11m39.17270891s) [noise=true , address=145.79.227.45:4229 , piece_id=GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.302170964 (469h11m39.173254009s) [noise=true , address=145.79.227.3:4150 , piece_id=OHGA4EA33AVEDCJLB6QSUJLD3W2TV6YNN6FBVFFKTTSF2B6XDPRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.302629947 (469h11m39.173712982s) [noise=true , address=145.79.227.19:4079 , piece_id=ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303079704 (469h11m39.174162749s) [noise=true , address=145.79.227.15:4875 , piece_id=FUMLT2ODHUE3ETSNWVKBYLSJXYVCHUW5KAJJE5AECNL3IAQYYCLQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303499804 (469h11m39.174582849s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303921386 (469h11m39.175004432s) [noise=true , address=145.79.227.47:4265 , piece_id=TVUXNR6K35A3CWFDN4JRKM4Q6RGBC4LP2XKPXJFN2AJL32ESEBPA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:41.172453460 (469h11m41.043536495s) [upload=411834]: Next returned nil segment Dec 25 04:40:41.172473798 (469h11m41.043556833s) [upload=411834]: Waiting for error group managing segments... Dec 25 04:40:41.483361556 (469h11m41.354444601s) [noise=true , address=145.79.227.37:4567 , piece_id=TZ4C57ISUI3CB3VOLTSM5W5QVDI7PVXV67PFJI3ZSVSXHTKPYPSA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.484283170 (469h11m41.355366205s) [noise=true , address=145.79.227.5:4221 , piece_id=C4SXUVTG2TZ4EIRBGL4TLOLBMYOAZ4KLIQSZOXKEOWAJBQQJLWSA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.515393699 (469h11m41.386476744s) [noise=true , address=145.79.227.51:4411 , piece_id=WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.526179387 (469h11m41.397262422s) [noise=true , address=45.91.116.5:4300 , piece_id=NTWC3RWYQQBBKPMXGUG44R2B5GWC2Z4LDFU3GTWOWCT3JIRIZOOQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.642380892 (469h11m41.513463937s) [noise=true , address=145.79.227.36:4517 , piece_id=HBQU7CMAO2KJS6ODW2WHICBAWQSH6TQQPOQSENKDEISWCWFN2CMA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.662326662 (469h11m41.533409707s) [noise=true , address=145.79.227.3:4150 , piece_id=OHGA4EA33AVEDCJLB6QSUJLD3W2TV6YNN6FBVFFKTTSF2B6XDPRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.758852130 (469h11m41.629935175s) [noise=true , address=145.79.227.47:4265 , piece_id=TVUXNR6K35A3CWFDN4JRKM4Q6RGBC4LP2XKPXJFN2AJL32ESEBPA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.823449082 (469h11m41.694532117s) [noise=true , address=145.79.227.17:4970 , piece_id=6DPKJ762IUN4A7UEZXLDERCVDW24QEKGEHFETAYZOKWQPVN2QJYA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.952636616 (469h11m41.823719661s) [noise=true , address=145.79.227.19:4079 , piece_id=ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.025177123 (469h11m41.896260168s) [noise=true , address=145.79.227.45:4229 , piece_id=GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.222526274 (469h11m42.093609319s) [noise=true , address=145.79.227.8:4331 , piece_id=ELLVNJFOVY5HSAHZDQYGQVO7OYHFZNFHCBJKHQFV5K6EV2M3JTRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.250024374 (469h11m42.121107409s) [noise=true , address=145.79.227.2:4114 , piece_id=5ISXZKQSH4CLLTF3ILGAXHV6STMLRUSZP6UATU2677X2FUPLBCRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.651787921 (469h11m42.522870966s) [noise=true , address=145.79.227.15:4875 , piece_id=FUMLT2ODHUE3ETSNWVKBYLSJXYVCHUW5KAJJE5AECNL3IAQYYCLQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.651814982 (469h11m42.522898027s) [seg_pos={0 0}, upload=411834]: Segment reached optimal threshold of 13 pieces. Dec 25 04:40:42.652052093 (469h11m42.523135129s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: write tcp 188.166.225.170:36650->77.90.31.15:4697: use of closed network connection; storage node overloaded, request limit: 2000 Dec 25 04:40:42.652069537 (469h11m42.523152572s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652177673 (469h11m42.523260708s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd): protocol: write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection; write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection Dec 25 04:40:42.652184456 (469h11m42.523267501s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652230243 (469h11m42.523313308s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq): protocol: write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection Dec 25 04:40:42.652240693 (469h11m42.523323778s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652268265 (469h11m42.52335131s) [seg_pos={0 0}, upload=411834]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 04:40:42.652354770 (469h11m42.523437815s) [upload=411834]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 04:40:42.652361693 (469h11m42.523444738s) [upload=411834]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 04:40:42.959704440 (469h11m42.830787485s) [upload=411834]: Done uploading segments... Dec 25 04:40:42.959728234 (469h11m42.83081127s) [upload=411834]: Upload finished. err: <nil> Duration: 5.919224056s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 5.670924155s trace: https://jaeger.dev.storj.io/trace/681cf842d2260d41 FAIL (2025-12-25 05:10:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, Piece ID: ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ) context deadline exceeded; piecestore: (Node ID: 1DT68m6J4PpDk3XUDhDaaqV46NLKVz4A23716JLFxEAh122GeR, Piece ID: WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA) context deadline exceeded; piecestore: (Node ID: 12wWYnhWHSqqAFgje4p9Cr7WYaJ6dJWsCqvrx8ZqVFoakjuFeka, Piece ID: GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q) context deadline exceeded ======LOGS====== nline: false Dec 25 04:40:39.039085560 (469h11m38.910168605s) [upload=411834]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 04:40:39.039095028 (469h11m38.910178073s) [upload=411834]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 04:40:39.297407613 (469h11m39.168490658s) [seg_pos={0 0}, upload=411834]: Begin upload segment... Dec 25 04:40:39.297471084 (469h11m39.168554119s) [seg_pos={0 0}, upload=411834]: Done begin upload segment. Dec 25 04:40:39.297511581 (469h11m39.168594616s) [noise=true , address=145.79.227.51:4411 , piece_id=WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.297663310 (469h11m39.168746345s) [noise=true , address=145.79.227.17:4970 , piece_id=6DPKJ762IUN4A7UEZXLDERCVDW24QEKGEHFETAYZOKWQPVN2QJYA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.298405574 (469h11m39.169488619s) [noise=true , address=145.79.227.2:4114 , piece_id=5ISXZKQSH4CLLTF3ILGAXHV6STMLRUSZP6UATU2677X2FUPLBCRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.298797961 (469h11m39.169881006s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299124392 (469h11m39.170207437s) [noise=true , address=45.91.116.5:4300 , piece_id=NTWC3RWYQQBBKPMXGUG44R2B5GWC2Z4LDFU3GTWOWCT3JIRIZOOQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299438650 (469h11m39.170521685s) [noise=true , address=145.79.227.5:4221 , piece_id=C4SXUVTG2TZ4EIRBGL4TLOLBMYOAZ4KLIQSZOXKEOWAJBQQJLWSA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299794709 (469h11m39.170877744s) [noise=true , address=145.79.227.37:4567 , piece_id=TZ4C57ISUI3CB3VOLTSM5W5QVDI7PVXV67PFJI3ZSVSXHTKPYPSA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.300142221 (469h11m39.171225256s) [noise=true , address=145.79.227.8:4331 , piece_id=ELLVNJFOVY5HSAHZDQYGQVO7OYHFZNFHCBJKHQFV5K6EV2M3JTRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.300506354 (469h11m39.171589389s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.301142795 (469h11m39.17222584s) [noise=true , address=145.79.227.36:4517 , piece_id=HBQU7CMAO2KJS6ODW2WHICBAWQSH6TQQPOQSENKDEISWCWFN2CMA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.301625835 (469h11m39.17270891s) [noise=true , address=145.79.227.45:4229 , piece_id=GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.302170964 (469h11m39.173254009s) [noise=true , address=145.79.227.3:4150 , piece_id=OHGA4EA33AVEDCJLB6QSUJLD3W2TV6YNN6FBVFFKTTSF2B6XDPRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.302629947 (469h11m39.173712982s) [noise=true , address=145.79.227.19:4079 , piece_id=ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303079704 (469h11m39.174162749s) [noise=true , address=145.79.227.15:4875 , piece_id=FUMLT2ODHUE3ETSNWVKBYLSJXYVCHUW5KAJJE5AECNL3IAQYYCLQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303499804 (469h11m39.174582849s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303921386 (469h11m39.175004432s) [noise=true , address=145.79.227.47:4265 , piece_id=TVUXNR6K35A3CWFDN4JRKM4Q6RGBC4LP2XKPXJFN2AJL32ESEBPA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:41.172453460 (469h11m41.043536495s) [upload=411834]: Next returned nil segment Dec 25 04:40:41.172473798 (469h11m41.043556833s) [upload=411834]: Waiting for error group managing segments... Dec 25 04:40:41.483361556 (469h11m41.354444601s) [noise=true , address=145.79.227.37:4567 , piece_id=TZ4C57ISUI3CB3VOLTSM5W5QVDI7PVXV67PFJI3ZSVSXHTKPYPSA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.484283170 (469h11m41.355366205s) [noise=true , address=145.79.227.5:4221 , piece_id=C4SXUVTG2TZ4EIRBGL4TLOLBMYOAZ4KLIQSZOXKEOWAJBQQJLWSA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.515393699 (469h11m41.386476744s) [noise=true , address=145.79.227.51:4411 , piece_id=WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.526179387 (469h11m41.397262422s) [noise=true , address=45.91.116.5:4300 , piece_id=NTWC3RWYQQBBKPMXGUG44R2B5GWC2Z4LDFU3GTWOWCT3JIRIZOOQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.642380892 (469h11m41.513463937s) [noise=true , address=145.79.227.36:4517 , piece_id=HBQU7CMAO2KJS6ODW2WHICBAWQSH6TQQPOQSENKDEISWCWFN2CMA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.662326662 (469h11m41.533409707s) [noise=true , address=145.79.227.3:4150 , piece_id=OHGA4EA33AVEDCJLB6QSUJLD3W2TV6YNN6FBVFFKTTSF2B6XDPRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.758852130 (469h11m41.629935175s) [noise=true , address=145.79.227.47:4265 , piece_id=TVUXNR6K35A3CWFDN4JRKM4Q6RGBC4LP2XKPXJFN2AJL32ESEBPA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.823449082 (469h11m41.694532117s) [noise=true , address=145.79.227.17:4970 , piece_id=6DPKJ762IUN4A7UEZXLDERCVDW24QEKGEHFETAYZOKWQPVN2QJYA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.952636616 (469h11m41.823719661s) [noise=true , address=145.79.227.19:4079 , piece_id=ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.025177123 (469h11m41.896260168s) [noise=true , address=145.79.227.45:4229 , piece_id=GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.222526274 (469h11m42.093609319s) [noise=true , address=145.79.227.8:4331 , piece_id=ELLVNJFOVY5HSAHZDQYGQVO7OYHFZNFHCBJKHQFV5K6EV2M3JTRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.250024374 (469h11m42.121107409s) [noise=true , address=145.79.227.2:4114 , piece_id=5ISXZKQSH4CLLTF3ILGAXHV6STMLRUSZP6UATU2677X2FUPLBCRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.651787921 (469h11m42.522870966s) [noise=true , address=145.79.227.15:4875 , piece_id=FUMLT2ODHUE3ETSNWVKBYLSJXYVCHUW5KAJJE5AECNL3IAQYYCLQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.651814982 (469h11m42.522898027s) [seg_pos={0 0}, upload=411834]: Segment reached optimal threshold of 13 pieces. Dec 25 04:40:42.652052093 (469h11m42.523135129s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: write tcp 188.166.225.170:36650->77.90.31.15:4697: use of closed network connection; storage node overloaded, request limit: 2000 Dec 25 04:40:42.652069537 (469h11m42.523152572s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652177673 (469h11m42.523260708s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd): protocol: write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection; write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection Dec 25 04:40:42.652184456 (469h11m42.523267501s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652230243 (469h11m42.523313308s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq): protocol: write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection Dec 25 04:40:42.652240693 (469h11m42.523323778s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652268265 (469h11m42.52335131s) [seg_pos={0 0}, upload=411834]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 04:40:42.652354770 (469h11m42.523437815s) [upload=411834]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 04:40:42.652361693 (469h11m42.523444738s) [upload=411834]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 04:40:42.959704440 (469h11m42.830787485s) [upload=411834]: Done uploading segments... Dec 25 04:40:42.959728234 (469h11m42.83081127s) [upload=411834]: Upload finished. err: <nil> Duration: 5.670969021s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 5.97141082s trace: https://jaeger.dev.storj.io/trace/681cf842d2260d41 FAIL (2025-12-25 05:10:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, Piece ID: ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ) context deadline exceeded; piecestore: (Node ID: 1DT68m6J4PpDk3XUDhDaaqV46NLKVz4A23716JLFxEAh122GeR, Piece ID: WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA) context deadline exceeded; piecestore: (Node ID: 12wWYnhWHSqqAFgje4p9Cr7WYaJ6dJWsCqvrx8ZqVFoakjuFeka, Piece ID: GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q) context deadline exceeded ======LOGS====== nline: false Dec 25 04:40:39.039085560 (469h11m38.910168605s) [upload=411834]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 04:40:39.039095028 (469h11m38.910178073s) [upload=411834]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 04:40:39.297407613 (469h11m39.168490658s) [seg_pos={0 0}, upload=411834]: Begin upload segment... Dec 25 04:40:39.297471084 (469h11m39.168554119s) [seg_pos={0 0}, upload=411834]: Done begin upload segment. Dec 25 04:40:39.297511581 (469h11m39.168594616s) [noise=true , address=145.79.227.51:4411 , piece_id=WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.297663310 (469h11m39.168746345s) [noise=true , address=145.79.227.17:4970 , piece_id=6DPKJ762IUN4A7UEZXLDERCVDW24QEKGEHFETAYZOKWQPVN2QJYA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.298405574 (469h11m39.169488619s) [noise=true , address=145.79.227.2:4114 , piece_id=5ISXZKQSH4CLLTF3ILGAXHV6STMLRUSZP6UATU2677X2FUPLBCRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.298797961 (469h11m39.169881006s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299124392 (469h11m39.170207437s) [noise=true , address=45.91.116.5:4300 , piece_id=NTWC3RWYQQBBKPMXGUG44R2B5GWC2Z4LDFU3GTWOWCT3JIRIZOOQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299438650 (469h11m39.170521685s) [noise=true , address=145.79.227.5:4221 , piece_id=C4SXUVTG2TZ4EIRBGL4TLOLBMYOAZ4KLIQSZOXKEOWAJBQQJLWSA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299794709 (469h11m39.170877744s) [noise=true , address=145.79.227.37:4567 , piece_id=TZ4C57ISUI3CB3VOLTSM5W5QVDI7PVXV67PFJI3ZSVSXHTKPYPSA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.300142221 (469h11m39.171225256s) [noise=true , address=145.79.227.8:4331 , piece_id=ELLVNJFOVY5HSAHZDQYGQVO7OYHFZNFHCBJKHQFV5K6EV2M3JTRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.300506354 (469h11m39.171589389s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.301142795 (469h11m39.17222584s) [noise=true , address=145.79.227.36:4517 , piece_id=HBQU7CMAO2KJS6ODW2WHICBAWQSH6TQQPOQSENKDEISWCWFN2CMA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.301625835 (469h11m39.17270891s) [noise=true , address=145.79.227.45:4229 , piece_id=GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.302170964 (469h11m39.173254009s) [noise=true , address=145.79.227.3:4150 , piece_id=OHGA4EA33AVEDCJLB6QSUJLD3W2TV6YNN6FBVFFKTTSF2B6XDPRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.302629947 (469h11m39.173712982s) [noise=true , address=145.79.227.19:4079 , piece_id=ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303079704 (469h11m39.174162749s) [noise=true , address=145.79.227.15:4875 , piece_id=FUMLT2ODHUE3ETSNWVKBYLSJXYVCHUW5KAJJE5AECNL3IAQYYCLQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303499804 (469h11m39.174582849s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303921386 (469h11m39.175004432s) [noise=true , address=145.79.227.47:4265 , piece_id=TVUXNR6K35A3CWFDN4JRKM4Q6RGBC4LP2XKPXJFN2AJL32ESEBPA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:41.172453460 (469h11m41.043536495s) [upload=411834]: Next returned nil segment Dec 25 04:40:41.172473798 (469h11m41.043556833s) [upload=411834]: Waiting for error group managing segments... Dec 25 04:40:41.483361556 (469h11m41.354444601s) [noise=true , address=145.79.227.37:4567 , piece_id=TZ4C57ISUI3CB3VOLTSM5W5QVDI7PVXV67PFJI3ZSVSXHTKPYPSA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.484283170 (469h11m41.355366205s) [noise=true , address=145.79.227.5:4221 , piece_id=C4SXUVTG2TZ4EIRBGL4TLOLBMYOAZ4KLIQSZOXKEOWAJBQQJLWSA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.515393699 (469h11m41.386476744s) [noise=true , address=145.79.227.51:4411 , piece_id=WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.526179387 (469h11m41.397262422s) [noise=true , address=45.91.116.5:4300 , piece_id=NTWC3RWYQQBBKPMXGUG44R2B5GWC2Z4LDFU3GTWOWCT3JIRIZOOQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.642380892 (469h11m41.513463937s) [noise=true , address=145.79.227.36:4517 , piece_id=HBQU7CMAO2KJS6ODW2WHICBAWQSH6TQQPOQSENKDEISWCWFN2CMA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.662326662 (469h11m41.533409707s) [noise=true , address=145.79.227.3:4150 , piece_id=OHGA4EA33AVEDCJLB6QSUJLD3W2TV6YNN6FBVFFKTTSF2B6XDPRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.758852130 (469h11m41.629935175s) [noise=true , address=145.79.227.47:4265 , piece_id=TVUXNR6K35A3CWFDN4JRKM4Q6RGBC4LP2XKPXJFN2AJL32ESEBPA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.823449082 (469h11m41.694532117s) [noise=true , address=145.79.227.17:4970 , piece_id=6DPKJ762IUN4A7UEZXLDERCVDW24QEKGEHFETAYZOKWQPVN2QJYA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.952636616 (469h11m41.823719661s) [noise=true , address=145.79.227.19:4079 , piece_id=ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.025177123 (469h11m41.896260168s) [noise=true , address=145.79.227.45:4229 , piece_id=GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.222526274 (469h11m42.093609319s) [noise=true , address=145.79.227.8:4331 , piece_id=ELLVNJFOVY5HSAHZDQYGQVO7OYHFZNFHCBJKHQFV5K6EV2M3JTRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.250024374 (469h11m42.121107409s) [noise=true , address=145.79.227.2:4114 , piece_id=5ISXZKQSH4CLLTF3ILGAXHV6STMLRUSZP6UATU2677X2FUPLBCRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.651787921 (469h11m42.522870966s) [noise=true , address=145.79.227.15:4875 , piece_id=FUMLT2ODHUE3ETSNWVKBYLSJXYVCHUW5KAJJE5AECNL3IAQYYCLQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.651814982 (469h11m42.522898027s) [seg_pos={0 0}, upload=411834]: Segment reached optimal threshold of 13 pieces. Dec 25 04:40:42.652052093 (469h11m42.523135129s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: write tcp 188.166.225.170:36650->77.90.31.15:4697: use of closed network connection; storage node overloaded, request limit: 2000 Dec 25 04:40:42.652069537 (469h11m42.523152572s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652177673 (469h11m42.523260708s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd): protocol: write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection; write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection Dec 25 04:40:42.652184456 (469h11m42.523267501s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652230243 (469h11m42.523313308s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq): protocol: write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection Dec 25 04:40:42.652240693 (469h11m42.523323778s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652268265 (469h11m42.52335131s) [seg_pos={0 0}, upload=411834]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 04:40:42.652354770 (469h11m42.523437815s) [upload=411834]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 04:40:42.652361693 (469h11m42.523444738s) [upload=411834]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 04:40:42.959704440 (469h11m42.830787485s) [upload=411834]: Done uploading segments... Dec 25 04:40:42.959728234 (469h11m42.83081127s) [upload=411834]: Upload finished. err: <nil> Duration: 5.971447289s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 6.633793493s trace: https://jaeger.dev.storj.io/trace/681cf842d2260d41 FAIL (2025-12-25 05:09:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, Piece ID: ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ) context deadline exceeded; piecestore: (Node ID: 1DT68m6J4PpDk3XUDhDaaqV46NLKVz4A23716JLFxEAh122GeR, Piece ID: WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA) context deadline exceeded; piecestore: (Node ID: 12wWYnhWHSqqAFgje4p9Cr7WYaJ6dJWsCqvrx8ZqVFoakjuFeka, Piece ID: GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q) context deadline exceeded ======LOGS====== nline: false Dec 25 04:40:39.039085560 (469h11m38.910168605s) [upload=411834]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 04:40:39.039095028 (469h11m38.910178073s) [upload=411834]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 04:40:39.297407613 (469h11m39.168490658s) [seg_pos={0 0}, upload=411834]: Begin upload segment... Dec 25 04:40:39.297471084 (469h11m39.168554119s) [seg_pos={0 0}, upload=411834]: Done begin upload segment. Dec 25 04:40:39.297511581 (469h11m39.168594616s) [noise=true , address=145.79.227.51:4411 , piece_id=WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.297663310 (469h11m39.168746345s) [noise=true , address=145.79.227.17:4970 , piece_id=6DPKJ762IUN4A7UEZXLDERCVDW24QEKGEHFETAYZOKWQPVN2QJYA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.298405574 (469h11m39.169488619s) [noise=true , address=145.79.227.2:4114 , piece_id=5ISXZKQSH4CLLTF3ILGAXHV6STMLRUSZP6UATU2677X2FUPLBCRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.298797961 (469h11m39.169881006s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299124392 (469h11m39.170207437s) [noise=true , address=45.91.116.5:4300 , piece_id=NTWC3RWYQQBBKPMXGUG44R2B5GWC2Z4LDFU3GTWOWCT3JIRIZOOQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299438650 (469h11m39.170521685s) [noise=true , address=145.79.227.5:4221 , piece_id=C4SXUVTG2TZ4EIRBGL4TLOLBMYOAZ4KLIQSZOXKEOWAJBQQJLWSA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299794709 (469h11m39.170877744s) [noise=true , address=145.79.227.37:4567 , piece_id=TZ4C57ISUI3CB3VOLTSM5W5QVDI7PVXV67PFJI3ZSVSXHTKPYPSA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.300142221 (469h11m39.171225256s) [noise=true , address=145.79.227.8:4331 , piece_id=ELLVNJFOVY5HSAHZDQYGQVO7OYHFZNFHCBJKHQFV5K6EV2M3JTRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.300506354 (469h11m39.171589389s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.301142795 (469h11m39.17222584s) [noise=true , address=145.79.227.36:4517 , piece_id=HBQU7CMAO2KJS6ODW2WHICBAWQSH6TQQPOQSENKDEISWCWFN2CMA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.301625835 (469h11m39.17270891s) [noise=true , address=145.79.227.45:4229 , piece_id=GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.302170964 (469h11m39.173254009s) [noise=true , address=145.79.227.3:4150 , piece_id=OHGA4EA33AVEDCJLB6QSUJLD3W2TV6YNN6FBVFFKTTSF2B6XDPRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.302629947 (469h11m39.173712982s) [noise=true , address=145.79.227.19:4079 , piece_id=ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303079704 (469h11m39.174162749s) [noise=true , address=145.79.227.15:4875 , piece_id=FUMLT2ODHUE3ETSNWVKBYLSJXYVCHUW5KAJJE5AECNL3IAQYYCLQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303499804 (469h11m39.174582849s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303921386 (469h11m39.175004432s) [noise=true , address=145.79.227.47:4265 , piece_id=TVUXNR6K35A3CWFDN4JRKM4Q6RGBC4LP2XKPXJFN2AJL32ESEBPA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:41.172453460 (469h11m41.043536495s) [upload=411834]: Next returned nil segment Dec 25 04:40:41.172473798 (469h11m41.043556833s) [upload=411834]: Waiting for error group managing segments... Dec 25 04:40:41.483361556 (469h11m41.354444601s) [noise=true , address=145.79.227.37:4567 , piece_id=TZ4C57ISUI3CB3VOLTSM5W5QVDI7PVXV67PFJI3ZSVSXHTKPYPSA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.484283170 (469h11m41.355366205s) [noise=true , address=145.79.227.5:4221 , piece_id=C4SXUVTG2TZ4EIRBGL4TLOLBMYOAZ4KLIQSZOXKEOWAJBQQJLWSA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.515393699 (469h11m41.386476744s) [noise=true , address=145.79.227.51:4411 , piece_id=WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.526179387 (469h11m41.397262422s) [noise=true , address=45.91.116.5:4300 , piece_id=NTWC3RWYQQBBKPMXGUG44R2B5GWC2Z4LDFU3GTWOWCT3JIRIZOOQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.642380892 (469h11m41.513463937s) [noise=true , address=145.79.227.36:4517 , piece_id=HBQU7CMAO2KJS6ODW2WHICBAWQSH6TQQPOQSENKDEISWCWFN2CMA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.662326662 (469h11m41.533409707s) [noise=true , address=145.79.227.3:4150 , piece_id=OHGA4EA33AVEDCJLB6QSUJLD3W2TV6YNN6FBVFFKTTSF2B6XDPRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.758852130 (469h11m41.629935175s) [noise=true , address=145.79.227.47:4265 , piece_id=TVUXNR6K35A3CWFDN4JRKM4Q6RGBC4LP2XKPXJFN2AJL32ESEBPA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.823449082 (469h11m41.694532117s) [noise=true , address=145.79.227.17:4970 , piece_id=6DPKJ762IUN4A7UEZXLDERCVDW24QEKGEHFETAYZOKWQPVN2QJYA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.952636616 (469h11m41.823719661s) [noise=true , address=145.79.227.19:4079 , piece_id=ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.025177123 (469h11m41.896260168s) [noise=true , address=145.79.227.45:4229 , piece_id=GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.222526274 (469h11m42.093609319s) [noise=true , address=145.79.227.8:4331 , piece_id=ELLVNJFOVY5HSAHZDQYGQVO7OYHFZNFHCBJKHQFV5K6EV2M3JTRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.250024374 (469h11m42.121107409s) [noise=true , address=145.79.227.2:4114 , piece_id=5ISXZKQSH4CLLTF3ILGAXHV6STMLRUSZP6UATU2677X2FUPLBCRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.651787921 (469h11m42.522870966s) [noise=true , address=145.79.227.15:4875 , piece_id=FUMLT2ODHUE3ETSNWVKBYLSJXYVCHUW5KAJJE5AECNL3IAQYYCLQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.651814982 (469h11m42.522898027s) [seg_pos={0 0}, upload=411834]: Segment reached optimal threshold of 13 pieces. Dec 25 04:40:42.652052093 (469h11m42.523135129s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: write tcp 188.166.225.170:36650->77.90.31.15:4697: use of closed network connection; storage node overloaded, request limit: 2000 Dec 25 04:40:42.652069537 (469h11m42.523152572s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652177673 (469h11m42.523260708s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd): protocol: write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection; write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection Dec 25 04:40:42.652184456 (469h11m42.523267501s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652230243 (469h11m42.523313308s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq): protocol: write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection Dec 25 04:40:42.652240693 (469h11m42.523323778s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652268265 (469h11m42.52335131s) [seg_pos={0 0}, upload=411834]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 04:40:42.652354770 (469h11m42.523437815s) [upload=411834]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 04:40:42.652361693 (469h11m42.523444738s) [upload=411834]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 04:40:42.959704440 (469h11m42.830787485s) [upload=411834]: Done uploading segments... Dec 25 04:40:42.959728234 (469h11m42.83081127s) [upload=411834]: Upload finished. err: <nil> Duration: 6.633818722s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 5.708975803s trace: https://jaeger.dev.storj.io/trace/681cf842d2260d41 FAIL (2025-12-25 05:08:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, Piece ID: ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ) context deadline exceeded; piecestore: (Node ID: 1DT68m6J4PpDk3XUDhDaaqV46NLKVz4A23716JLFxEAh122GeR, Piece ID: WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA) context deadline exceeded; piecestore: (Node ID: 12wWYnhWHSqqAFgje4p9Cr7WYaJ6dJWsCqvrx8ZqVFoakjuFeka, Piece ID: GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q) context deadline exceeded ======LOGS====== nline: false Dec 25 04:40:39.039085560 (469h11m38.910168605s) [upload=411834]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 04:40:39.039095028 (469h11m38.910178073s) [upload=411834]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 04:40:39.297407613 (469h11m39.168490658s) [seg_pos={0 0}, upload=411834]: Begin upload segment... Dec 25 04:40:39.297471084 (469h11m39.168554119s) [seg_pos={0 0}, upload=411834]: Done begin upload segment. Dec 25 04:40:39.297511581 (469h11m39.168594616s) [noise=true , address=145.79.227.51:4411 , piece_id=WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.297663310 (469h11m39.168746345s) [noise=true , address=145.79.227.17:4970 , piece_id=6DPKJ762IUN4A7UEZXLDERCVDW24QEKGEHFETAYZOKWQPVN2QJYA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.298405574 (469h11m39.169488619s) [noise=true , address=145.79.227.2:4114 , piece_id=5ISXZKQSH4CLLTF3ILGAXHV6STMLRUSZP6UATU2677X2FUPLBCRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.298797961 (469h11m39.169881006s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299124392 (469h11m39.170207437s) [noise=true , address=45.91.116.5:4300 , piece_id=NTWC3RWYQQBBKPMXGUG44R2B5GWC2Z4LDFU3GTWOWCT3JIRIZOOQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299438650 (469h11m39.170521685s) [noise=true , address=145.79.227.5:4221 , piece_id=C4SXUVTG2TZ4EIRBGL4TLOLBMYOAZ4KLIQSZOXKEOWAJBQQJLWSA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299794709 (469h11m39.170877744s) [noise=true , address=145.79.227.37:4567 , piece_id=TZ4C57ISUI3CB3VOLTSM5W5QVDI7PVXV67PFJI3ZSVSXHTKPYPSA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.300142221 (469h11m39.171225256s) [noise=true , address=145.79.227.8:4331 , piece_id=ELLVNJFOVY5HSAHZDQYGQVO7OYHFZNFHCBJKHQFV5K6EV2M3JTRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.300506354 (469h11m39.171589389s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.301142795 (469h11m39.17222584s) [noise=true , address=145.79.227.36:4517 , piece_id=HBQU7CMAO2KJS6ODW2WHICBAWQSH6TQQPOQSENKDEISWCWFN2CMA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.301625835 (469h11m39.17270891s) [noise=true , address=145.79.227.45:4229 , piece_id=GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.302170964 (469h11m39.173254009s) [noise=true , address=145.79.227.3:4150 , piece_id=OHGA4EA33AVEDCJLB6QSUJLD3W2TV6YNN6FBVFFKTTSF2B6XDPRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.302629947 (469h11m39.173712982s) [noise=true , address=145.79.227.19:4079 , piece_id=ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303079704 (469h11m39.174162749s) [noise=true , address=145.79.227.15:4875 , piece_id=FUMLT2ODHUE3ETSNWVKBYLSJXYVCHUW5KAJJE5AECNL3IAQYYCLQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303499804 (469h11m39.174582849s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303921386 (469h11m39.175004432s) [noise=true , address=145.79.227.47:4265 , piece_id=TVUXNR6K35A3CWFDN4JRKM4Q6RGBC4LP2XKPXJFN2AJL32ESEBPA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:41.172453460 (469h11m41.043536495s) [upload=411834]: Next returned nil segment Dec 25 04:40:41.172473798 (469h11m41.043556833s) [upload=411834]: Waiting for error group managing segments... Dec 25 04:40:41.483361556 (469h11m41.354444601s) [noise=true , address=145.79.227.37:4567 , piece_id=TZ4C57ISUI3CB3VOLTSM5W5QVDI7PVXV67PFJI3ZSVSXHTKPYPSA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.484283170 (469h11m41.355366205s) [noise=true , address=145.79.227.5:4221 , piece_id=C4SXUVTG2TZ4EIRBGL4TLOLBMYOAZ4KLIQSZOXKEOWAJBQQJLWSA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.515393699 (469h11m41.386476744s) [noise=true , address=145.79.227.51:4411 , piece_id=WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.526179387 (469h11m41.397262422s) [noise=true , address=45.91.116.5:4300 , piece_id=NTWC3RWYQQBBKPMXGUG44R2B5GWC2Z4LDFU3GTWOWCT3JIRIZOOQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.642380892 (469h11m41.513463937s) [noise=true , address=145.79.227.36:4517 , piece_id=HBQU7CMAO2KJS6ODW2WHICBAWQSH6TQQPOQSENKDEISWCWFN2CMA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.662326662 (469h11m41.533409707s) [noise=true , address=145.79.227.3:4150 , piece_id=OHGA4EA33AVEDCJLB6QSUJLD3W2TV6YNN6FBVFFKTTSF2B6XDPRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.758852130 (469h11m41.629935175s) [noise=true , address=145.79.227.47:4265 , piece_id=TVUXNR6K35A3CWFDN4JRKM4Q6RGBC4LP2XKPXJFN2AJL32ESEBPA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.823449082 (469h11m41.694532117s) [noise=true , address=145.79.227.17:4970 , piece_id=6DPKJ762IUN4A7UEZXLDERCVDW24QEKGEHFETAYZOKWQPVN2QJYA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.952636616 (469h11m41.823719661s) [noise=true , address=145.79.227.19:4079 , piece_id=ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.025177123 (469h11m41.896260168s) [noise=true , address=145.79.227.45:4229 , piece_id=GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.222526274 (469h11m42.093609319s) [noise=true , address=145.79.227.8:4331 , piece_id=ELLVNJFOVY5HSAHZDQYGQVO7OYHFZNFHCBJKHQFV5K6EV2M3JTRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.250024374 (469h11m42.121107409s) [noise=true , address=145.79.227.2:4114 , piece_id=5ISXZKQSH4CLLTF3ILGAXHV6STMLRUSZP6UATU2677X2FUPLBCRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.651787921 (469h11m42.522870966s) [noise=true , address=145.79.227.15:4875 , piece_id=FUMLT2ODHUE3ETSNWVKBYLSJXYVCHUW5KAJJE5AECNL3IAQYYCLQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.651814982 (469h11m42.522898027s) [seg_pos={0 0}, upload=411834]: Segment reached optimal threshold of 13 pieces. Dec 25 04:40:42.652052093 (469h11m42.523135129s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: write tcp 188.166.225.170:36650->77.90.31.15:4697: use of closed network connection; storage node overloaded, request limit: 2000 Dec 25 04:40:42.652069537 (469h11m42.523152572s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652177673 (469h11m42.523260708s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd): protocol: write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection; write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection Dec 25 04:40:42.652184456 (469h11m42.523267501s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652230243 (469h11m42.523313308s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq): protocol: write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection Dec 25 04:40:42.652240693 (469h11m42.523323778s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652268265 (469h11m42.52335131s) [seg_pos={0 0}, upload=411834]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 04:40:42.652354770 (469h11m42.523437815s) [upload=411834]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 04:40:42.652361693 (469h11m42.523444738s) [upload=411834]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 04:40:42.959704440 (469h11m42.830787485s) [upload=411834]: Done uploading segments... Dec 25 04:40:42.959728234 (469h11m42.83081127s) [upload=411834]: Upload finished. err: <nil> Duration: 5.709001872s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 6.184400908s trace: https://jaeger.dev.storj.io/trace/681cf842d2260d41 FAIL (2025-12-25 05:07:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, Piece ID: ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ) context deadline exceeded; piecestore: (Node ID: 1DT68m6J4PpDk3XUDhDaaqV46NLKVz4A23716JLFxEAh122GeR, Piece ID: WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA) context deadline exceeded; piecestore: (Node ID: 12wWYnhWHSqqAFgje4p9Cr7WYaJ6dJWsCqvrx8ZqVFoakjuFeka, Piece ID: GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q) context deadline exceeded ======LOGS====== nline: false Dec 25 04:40:39.039085560 (469h11m38.910168605s) [upload=411834]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 04:40:39.039095028 (469h11m38.910178073s) [upload=411834]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 04:40:39.297407613 (469h11m39.168490658s) [seg_pos={0 0}, upload=411834]: Begin upload segment... Dec 25 04:40:39.297471084 (469h11m39.168554119s) [seg_pos={0 0}, upload=411834]: Done begin upload segment. Dec 25 04:40:39.297511581 (469h11m39.168594616s) [noise=true , address=145.79.227.51:4411 , piece_id=WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.297663310 (469h11m39.168746345s) [noise=true , address=145.79.227.17:4970 , piece_id=6DPKJ762IUN4A7UEZXLDERCVDW24QEKGEHFETAYZOKWQPVN2QJYA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.298405574 (469h11m39.169488619s) [noise=true , address=145.79.227.2:4114 , piece_id=5ISXZKQSH4CLLTF3ILGAXHV6STMLRUSZP6UATU2677X2FUPLBCRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.298797961 (469h11m39.169881006s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299124392 (469h11m39.170207437s) [noise=true , address=45.91.116.5:4300 , piece_id=NTWC3RWYQQBBKPMXGUG44R2B5GWC2Z4LDFU3GTWOWCT3JIRIZOOQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299438650 (469h11m39.170521685s) [noise=true , address=145.79.227.5:4221 , piece_id=C4SXUVTG2TZ4EIRBGL4TLOLBMYOAZ4KLIQSZOXKEOWAJBQQJLWSA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299794709 (469h11m39.170877744s) [noise=true , address=145.79.227.37:4567 , piece_id=TZ4C57ISUI3CB3VOLTSM5W5QVDI7PVXV67PFJI3ZSVSXHTKPYPSA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.300142221 (469h11m39.171225256s) [noise=true , address=145.79.227.8:4331 , piece_id=ELLVNJFOVY5HSAHZDQYGQVO7OYHFZNFHCBJKHQFV5K6EV2M3JTRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.300506354 (469h11m39.171589389s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.301142795 (469h11m39.17222584s) [noise=true , address=145.79.227.36:4517 , piece_id=HBQU7CMAO2KJS6ODW2WHICBAWQSH6TQQPOQSENKDEISWCWFN2CMA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.301625835 (469h11m39.17270891s) [noise=true , address=145.79.227.45:4229 , piece_id=GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.302170964 (469h11m39.173254009s) [noise=true , address=145.79.227.3:4150 , piece_id=OHGA4EA33AVEDCJLB6QSUJLD3W2TV6YNN6FBVFFKTTSF2B6XDPRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.302629947 (469h11m39.173712982s) [noise=true , address=145.79.227.19:4079 , piece_id=ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303079704 (469h11m39.174162749s) [noise=true , address=145.79.227.15:4875 , piece_id=FUMLT2ODHUE3ETSNWVKBYLSJXYVCHUW5KAJJE5AECNL3IAQYYCLQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303499804 (469h11m39.174582849s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303921386 (469h11m39.175004432s) [noise=true , address=145.79.227.47:4265 , piece_id=TVUXNR6K35A3CWFDN4JRKM4Q6RGBC4LP2XKPXJFN2AJL32ESEBPA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:41.172453460 (469h11m41.043536495s) [upload=411834]: Next returned nil segment Dec 25 04:40:41.172473798 (469h11m41.043556833s) [upload=411834]: Waiting for error group managing segments... Dec 25 04:40:41.483361556 (469h11m41.354444601s) [noise=true , address=145.79.227.37:4567 , piece_id=TZ4C57ISUI3CB3VOLTSM5W5QVDI7PVXV67PFJI3ZSVSXHTKPYPSA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.484283170 (469h11m41.355366205s) [noise=true , address=145.79.227.5:4221 , piece_id=C4SXUVTG2TZ4EIRBGL4TLOLBMYOAZ4KLIQSZOXKEOWAJBQQJLWSA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.515393699 (469h11m41.386476744s) [noise=true , address=145.79.227.51:4411 , piece_id=WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.526179387 (469h11m41.397262422s) [noise=true , address=45.91.116.5:4300 , piece_id=NTWC3RWYQQBBKPMXGUG44R2B5GWC2Z4LDFU3GTWOWCT3JIRIZOOQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.642380892 (469h11m41.513463937s) [noise=true , address=145.79.227.36:4517 , piece_id=HBQU7CMAO2KJS6ODW2WHICBAWQSH6TQQPOQSENKDEISWCWFN2CMA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.662326662 (469h11m41.533409707s) [noise=true , address=145.79.227.3:4150 , piece_id=OHGA4EA33AVEDCJLB6QSUJLD3W2TV6YNN6FBVFFKTTSF2B6XDPRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.758852130 (469h11m41.629935175s) [noise=true , address=145.79.227.47:4265 , piece_id=TVUXNR6K35A3CWFDN4JRKM4Q6RGBC4LP2XKPXJFN2AJL32ESEBPA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.823449082 (469h11m41.694532117s) [noise=true , address=145.79.227.17:4970 , piece_id=6DPKJ762IUN4A7UEZXLDERCVDW24QEKGEHFETAYZOKWQPVN2QJYA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.952636616 (469h11m41.823719661s) [noise=true , address=145.79.227.19:4079 , piece_id=ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.025177123 (469h11m41.896260168s) [noise=true , address=145.79.227.45:4229 , piece_id=GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.222526274 (469h11m42.093609319s) [noise=true , address=145.79.227.8:4331 , piece_id=ELLVNJFOVY5HSAHZDQYGQVO7OYHFZNFHCBJKHQFV5K6EV2M3JTRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.250024374 (469h11m42.121107409s) [noise=true , address=145.79.227.2:4114 , piece_id=5ISXZKQSH4CLLTF3ILGAXHV6STMLRUSZP6UATU2677X2FUPLBCRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.651787921 (469h11m42.522870966s) [noise=true , address=145.79.227.15:4875 , piece_id=FUMLT2ODHUE3ETSNWVKBYLSJXYVCHUW5KAJJE5AECNL3IAQYYCLQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.651814982 (469h11m42.522898027s) [seg_pos={0 0}, upload=411834]: Segment reached optimal threshold of 13 pieces. Dec 25 04:40:42.652052093 (469h11m42.523135129s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: write tcp 188.166.225.170:36650->77.90.31.15:4697: use of closed network connection; storage node overloaded, request limit: 2000 Dec 25 04:40:42.652069537 (469h11m42.523152572s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652177673 (469h11m42.523260708s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd): protocol: write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection; write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection Dec 25 04:40:42.652184456 (469h11m42.523267501s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652230243 (469h11m42.523313308s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq): protocol: write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection Dec 25 04:40:42.652240693 (469h11m42.523323778s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652268265 (469h11m42.52335131s) [seg_pos={0 0}, upload=411834]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 04:40:42.652354770 (469h11m42.523437815s) [upload=411834]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 04:40:42.652361693 (469h11m42.523444738s) [upload=411834]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 04:40:42.959704440 (469h11m42.830787485s) [upload=411834]: Done uploading segments... Dec 25 04:40:42.959728234 (469h11m42.83081127s) [upload=411834]: Upload finished. err: <nil> Duration: 6.184421267s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 5.728006373s trace: https://jaeger.dev.storj.io/trace/681cf842d2260d41 FAIL (2025-12-25 05:06:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, Piece ID: ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ) context deadline exceeded; piecestore: (Node ID: 1DT68m6J4PpDk3XUDhDaaqV46NLKVz4A23716JLFxEAh122GeR, Piece ID: WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA) context deadline exceeded; piecestore: (Node ID: 12wWYnhWHSqqAFgje4p9Cr7WYaJ6dJWsCqvrx8ZqVFoakjuFeka, Piece ID: GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q) context deadline exceeded ======LOGS====== nline: false Dec 25 04:40:39.039085560 (469h11m38.910168605s) [upload=411834]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 04:40:39.039095028 (469h11m38.910178073s) [upload=411834]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 04:40:39.297407613 (469h11m39.168490658s) [seg_pos={0 0}, upload=411834]: Begin upload segment... Dec 25 04:40:39.297471084 (469h11m39.168554119s) [seg_pos={0 0}, upload=411834]: Done begin upload segment. Dec 25 04:40:39.297511581 (469h11m39.168594616s) [noise=true , address=145.79.227.51:4411 , piece_id=WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.297663310 (469h11m39.168746345s) [noise=true , address=145.79.227.17:4970 , piece_id=6DPKJ762IUN4A7UEZXLDERCVDW24QEKGEHFETAYZOKWQPVN2QJYA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.298405574 (469h11m39.169488619s) [noise=true , address=145.79.227.2:4114 , piece_id=5ISXZKQSH4CLLTF3ILGAXHV6STMLRUSZP6UATU2677X2FUPLBCRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.298797961 (469h11m39.169881006s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299124392 (469h11m39.170207437s) [noise=true , address=45.91.116.5:4300 , piece_id=NTWC3RWYQQBBKPMXGUG44R2B5GWC2Z4LDFU3GTWOWCT3JIRIZOOQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299438650 (469h11m39.170521685s) [noise=true , address=145.79.227.5:4221 , piece_id=C4SXUVTG2TZ4EIRBGL4TLOLBMYOAZ4KLIQSZOXKEOWAJBQQJLWSA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299794709 (469h11m39.170877744s) [noise=true , address=145.79.227.37:4567 , piece_id=TZ4C57ISUI3CB3VOLTSM5W5QVDI7PVXV67PFJI3ZSVSXHTKPYPSA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.300142221 (469h11m39.171225256s) [noise=true , address=145.79.227.8:4331 , piece_id=ELLVNJFOVY5HSAHZDQYGQVO7OYHFZNFHCBJKHQFV5K6EV2M3JTRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.300506354 (469h11m39.171589389s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.301142795 (469h11m39.17222584s) [noise=true , address=145.79.227.36:4517 , piece_id=HBQU7CMAO2KJS6ODW2WHICBAWQSH6TQQPOQSENKDEISWCWFN2CMA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.301625835 (469h11m39.17270891s) [noise=true , address=145.79.227.45:4229 , piece_id=GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.302170964 (469h11m39.173254009s) [noise=true , address=145.79.227.3:4150 , piece_id=OHGA4EA33AVEDCJLB6QSUJLD3W2TV6YNN6FBVFFKTTSF2B6XDPRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.302629947 (469h11m39.173712982s) [noise=true , address=145.79.227.19:4079 , piece_id=ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303079704 (469h11m39.174162749s) [noise=true , address=145.79.227.15:4875 , piece_id=FUMLT2ODHUE3ETSNWVKBYLSJXYVCHUW5KAJJE5AECNL3IAQYYCLQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303499804 (469h11m39.174582849s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303921386 (469h11m39.175004432s) [noise=true , address=145.79.227.47:4265 , piece_id=TVUXNR6K35A3CWFDN4JRKM4Q6RGBC4LP2XKPXJFN2AJL32ESEBPA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:41.172453460 (469h11m41.043536495s) [upload=411834]: Next returned nil segment Dec 25 04:40:41.172473798 (469h11m41.043556833s) [upload=411834]: Waiting for error group managing segments... Dec 25 04:40:41.483361556 (469h11m41.354444601s) [noise=true , address=145.79.227.37:4567 , piece_id=TZ4C57ISUI3CB3VOLTSM5W5QVDI7PVXV67PFJI3ZSVSXHTKPYPSA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.484283170 (469h11m41.355366205s) [noise=true , address=145.79.227.5:4221 , piece_id=C4SXUVTG2TZ4EIRBGL4TLOLBMYOAZ4KLIQSZOXKEOWAJBQQJLWSA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.515393699 (469h11m41.386476744s) [noise=true , address=145.79.227.51:4411 , piece_id=WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.526179387 (469h11m41.397262422s) [noise=true , address=45.91.116.5:4300 , piece_id=NTWC3RWYQQBBKPMXGUG44R2B5GWC2Z4LDFU3GTWOWCT3JIRIZOOQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.642380892 (469h11m41.513463937s) [noise=true , address=145.79.227.36:4517 , piece_id=HBQU7CMAO2KJS6ODW2WHICBAWQSH6TQQPOQSENKDEISWCWFN2CMA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.662326662 (469h11m41.533409707s) [noise=true , address=145.79.227.3:4150 , piece_id=OHGA4EA33AVEDCJLB6QSUJLD3W2TV6YNN6FBVFFKTTSF2B6XDPRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.758852130 (469h11m41.629935175s) [noise=true , address=145.79.227.47:4265 , piece_id=TVUXNR6K35A3CWFDN4JRKM4Q6RGBC4LP2XKPXJFN2AJL32ESEBPA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.823449082 (469h11m41.694532117s) [noise=true , address=145.79.227.17:4970 , piece_id=6DPKJ762IUN4A7UEZXLDERCVDW24QEKGEHFETAYZOKWQPVN2QJYA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.952636616 (469h11m41.823719661s) [noise=true , address=145.79.227.19:4079 , piece_id=ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.025177123 (469h11m41.896260168s) [noise=true , address=145.79.227.45:4229 , piece_id=GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.222526274 (469h11m42.093609319s) [noise=true , address=145.79.227.8:4331 , piece_id=ELLVNJFOVY5HSAHZDQYGQVO7OYHFZNFHCBJKHQFV5K6EV2M3JTRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.250024374 (469h11m42.121107409s) [noise=true , address=145.79.227.2:4114 , piece_id=5ISXZKQSH4CLLTF3ILGAXHV6STMLRUSZP6UATU2677X2FUPLBCRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.651787921 (469h11m42.522870966s) [noise=true , address=145.79.227.15:4875 , piece_id=FUMLT2ODHUE3ETSNWVKBYLSJXYVCHUW5KAJJE5AECNL3IAQYYCLQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.651814982 (469h11m42.522898027s) [seg_pos={0 0}, upload=411834]: Segment reached optimal threshold of 13 pieces. Dec 25 04:40:42.652052093 (469h11m42.523135129s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: write tcp 188.166.225.170:36650->77.90.31.15:4697: use of closed network connection; storage node overloaded, request limit: 2000 Dec 25 04:40:42.652069537 (469h11m42.523152572s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652177673 (469h11m42.523260708s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd): protocol: write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection; write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection Dec 25 04:40:42.652184456 (469h11m42.523267501s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652230243 (469h11m42.523313308s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq): protocol: write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection Dec 25 04:40:42.652240693 (469h11m42.523323778s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652268265 (469h11m42.52335131s) [seg_pos={0 0}, upload=411834]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 04:40:42.652354770 (469h11m42.523437815s) [upload=411834]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 04:40:42.652361693 (469h11m42.523444738s) [upload=411834]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 04:40:42.959704440 (469h11m42.830787485s) [upload=411834]: Done uploading segments... Dec 25 04:40:42.959728234 (469h11m42.83081127s) [upload=411834]: Upload finished. err: <nil> Duration: 5.728039426s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 5.891318368s trace: https://jaeger.dev.storj.io/trace/681cf842d2260d41 FAIL (2025-12-25 05:05:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, Piece ID: ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ) context deadline exceeded; piecestore: (Node ID: 1DT68m6J4PpDk3XUDhDaaqV46NLKVz4A23716JLFxEAh122GeR, Piece ID: WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA) context deadline exceeded; piecestore: (Node ID: 12wWYnhWHSqqAFgje4p9Cr7WYaJ6dJWsCqvrx8ZqVFoakjuFeka, Piece ID: GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q) context deadline exceeded ======LOGS====== nline: false Dec 25 04:40:39.039085560 (469h11m38.910168605s) [upload=411834]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 04:40:39.039095028 (469h11m38.910178073s) [upload=411834]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 04:40:39.297407613 (469h11m39.168490658s) [seg_pos={0 0}, upload=411834]: Begin upload segment... Dec 25 04:40:39.297471084 (469h11m39.168554119s) [seg_pos={0 0}, upload=411834]: Done begin upload segment. Dec 25 04:40:39.297511581 (469h11m39.168594616s) [noise=true , address=145.79.227.51:4411 , piece_id=WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.297663310 (469h11m39.168746345s) [noise=true , address=145.79.227.17:4970 , piece_id=6DPKJ762IUN4A7UEZXLDERCVDW24QEKGEHFETAYZOKWQPVN2QJYA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.298405574 (469h11m39.169488619s) [noise=true , address=145.79.227.2:4114 , piece_id=5ISXZKQSH4CLLTF3ILGAXHV6STMLRUSZP6UATU2677X2FUPLBCRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.298797961 (469h11m39.169881006s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299124392 (469h11m39.170207437s) [noise=true , address=45.91.116.5:4300 , piece_id=NTWC3RWYQQBBKPMXGUG44R2B5GWC2Z4LDFU3GTWOWCT3JIRIZOOQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299438650 (469h11m39.170521685s) [noise=true , address=145.79.227.5:4221 , piece_id=C4SXUVTG2TZ4EIRBGL4TLOLBMYOAZ4KLIQSZOXKEOWAJBQQJLWSA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299794709 (469h11m39.170877744s) [noise=true , address=145.79.227.37:4567 , piece_id=TZ4C57ISUI3CB3VOLTSM5W5QVDI7PVXV67PFJI3ZSVSXHTKPYPSA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.300142221 (469h11m39.171225256s) [noise=true , address=145.79.227.8:4331 , piece_id=ELLVNJFOVY5HSAHZDQYGQVO7OYHFZNFHCBJKHQFV5K6EV2M3JTRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.300506354 (469h11m39.171589389s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.301142795 (469h11m39.17222584s) [noise=true , address=145.79.227.36:4517 , piece_id=HBQU7CMAO2KJS6ODW2WHICBAWQSH6TQQPOQSENKDEISWCWFN2CMA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.301625835 (469h11m39.17270891s) [noise=true , address=145.79.227.45:4229 , piece_id=GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.302170964 (469h11m39.173254009s) [noise=true , address=145.79.227.3:4150 , piece_id=OHGA4EA33AVEDCJLB6QSUJLD3W2TV6YNN6FBVFFKTTSF2B6XDPRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.302629947 (469h11m39.173712982s) [noise=true , address=145.79.227.19:4079 , piece_id=ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303079704 (469h11m39.174162749s) [noise=true , address=145.79.227.15:4875 , piece_id=FUMLT2ODHUE3ETSNWVKBYLSJXYVCHUW5KAJJE5AECNL3IAQYYCLQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303499804 (469h11m39.174582849s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303921386 (469h11m39.175004432s) [noise=true , address=145.79.227.47:4265 , piece_id=TVUXNR6K35A3CWFDN4JRKM4Q6RGBC4LP2XKPXJFN2AJL32ESEBPA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:41.172453460 (469h11m41.043536495s) [upload=411834]: Next returned nil segment Dec 25 04:40:41.172473798 (469h11m41.043556833s) [upload=411834]: Waiting for error group managing segments... Dec 25 04:40:41.483361556 (469h11m41.354444601s) [noise=true , address=145.79.227.37:4567 , piece_id=TZ4C57ISUI3CB3VOLTSM5W5QVDI7PVXV67PFJI3ZSVSXHTKPYPSA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.484283170 (469h11m41.355366205s) [noise=true , address=145.79.227.5:4221 , piece_id=C4SXUVTG2TZ4EIRBGL4TLOLBMYOAZ4KLIQSZOXKEOWAJBQQJLWSA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.515393699 (469h11m41.386476744s) [noise=true , address=145.79.227.51:4411 , piece_id=WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.526179387 (469h11m41.397262422s) [noise=true , address=45.91.116.5:4300 , piece_id=NTWC3RWYQQBBKPMXGUG44R2B5GWC2Z4LDFU3GTWOWCT3JIRIZOOQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.642380892 (469h11m41.513463937s) [noise=true , address=145.79.227.36:4517 , piece_id=HBQU7CMAO2KJS6ODW2WHICBAWQSH6TQQPOQSENKDEISWCWFN2CMA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.662326662 (469h11m41.533409707s) [noise=true , address=145.79.227.3:4150 , piece_id=OHGA4EA33AVEDCJLB6QSUJLD3W2TV6YNN6FBVFFKTTSF2B6XDPRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.758852130 (469h11m41.629935175s) [noise=true , address=145.79.227.47:4265 , piece_id=TVUXNR6K35A3CWFDN4JRKM4Q6RGBC4LP2XKPXJFN2AJL32ESEBPA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.823449082 (469h11m41.694532117s) [noise=true , address=145.79.227.17:4970 , piece_id=6DPKJ762IUN4A7UEZXLDERCVDW24QEKGEHFETAYZOKWQPVN2QJYA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.952636616 (469h11m41.823719661s) [noise=true , address=145.79.227.19:4079 , piece_id=ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.025177123 (469h11m41.896260168s) [noise=true , address=145.79.227.45:4229 , piece_id=GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.222526274 (469h11m42.093609319s) [noise=true , address=145.79.227.8:4331 , piece_id=ELLVNJFOVY5HSAHZDQYGQVO7OYHFZNFHCBJKHQFV5K6EV2M3JTRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.250024374 (469h11m42.121107409s) [noise=true , address=145.79.227.2:4114 , piece_id=5ISXZKQSH4CLLTF3ILGAXHV6STMLRUSZP6UATU2677X2FUPLBCRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.651787921 (469h11m42.522870966s) [noise=true , address=145.79.227.15:4875 , piece_id=FUMLT2ODHUE3ETSNWVKBYLSJXYVCHUW5KAJJE5AECNL3IAQYYCLQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.651814982 (469h11m42.522898027s) [seg_pos={0 0}, upload=411834]: Segment reached optimal threshold of 13 pieces. Dec 25 04:40:42.652052093 (469h11m42.523135129s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: write tcp 188.166.225.170:36650->77.90.31.15:4697: use of closed network connection; storage node overloaded, request limit: 2000 Dec 25 04:40:42.652069537 (469h11m42.523152572s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652177673 (469h11m42.523260708s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd): protocol: write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection; write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection Dec 25 04:40:42.652184456 (469h11m42.523267501s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652230243 (469h11m42.523313308s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq): protocol: write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection Dec 25 04:40:42.652240693 (469h11m42.523323778s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652268265 (469h11m42.52335131s) [seg_pos={0 0}, upload=411834]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 04:40:42.652354770 (469h11m42.523437815s) [upload=411834]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 04:40:42.652361693 (469h11m42.523444738s) [upload=411834]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 04:40:42.959704440 (469h11m42.830787485s) [upload=411834]: Done uploading segments... Dec 25 04:40:42.959728234 (469h11m42.83081127s) [upload=411834]: Upload finished. err: <nil> Duration: 5.891385956s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 6.076164827s trace: https://jaeger.dev.storj.io/trace/681cf842d2260d41 FAIL (2025-12-25 05:05:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, Piece ID: ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ) context deadline exceeded; piecestore: (Node ID: 1DT68m6J4PpDk3XUDhDaaqV46NLKVz4A23716JLFxEAh122GeR, Piece ID: WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA) context deadline exceeded; piecestore: (Node ID: 12wWYnhWHSqqAFgje4p9Cr7WYaJ6dJWsCqvrx8ZqVFoakjuFeka, Piece ID: GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q) context deadline exceeded ======LOGS====== nline: false Dec 25 04:40:39.039085560 (469h11m38.910168605s) [upload=411834]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 04:40:39.039095028 (469h11m38.910178073s) [upload=411834]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 04:40:39.297407613 (469h11m39.168490658s) [seg_pos={0 0}, upload=411834]: Begin upload segment... Dec 25 04:40:39.297471084 (469h11m39.168554119s) [seg_pos={0 0}, upload=411834]: Done begin upload segment. Dec 25 04:40:39.297511581 (469h11m39.168594616s) [noise=true , address=145.79.227.51:4411 , piece_id=WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.297663310 (469h11m39.168746345s) [noise=true , address=145.79.227.17:4970 , piece_id=6DPKJ762IUN4A7UEZXLDERCVDW24QEKGEHFETAYZOKWQPVN2QJYA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.298405574 (469h11m39.169488619s) [noise=true , address=145.79.227.2:4114 , piece_id=5ISXZKQSH4CLLTF3ILGAXHV6STMLRUSZP6UATU2677X2FUPLBCRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.298797961 (469h11m39.169881006s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299124392 (469h11m39.170207437s) [noise=true , address=45.91.116.5:4300 , piece_id=NTWC3RWYQQBBKPMXGUG44R2B5GWC2Z4LDFU3GTWOWCT3JIRIZOOQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299438650 (469h11m39.170521685s) [noise=true , address=145.79.227.5:4221 , piece_id=C4SXUVTG2TZ4EIRBGL4TLOLBMYOAZ4KLIQSZOXKEOWAJBQQJLWSA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299794709 (469h11m39.170877744s) [noise=true , address=145.79.227.37:4567 , piece_id=TZ4C57ISUI3CB3VOLTSM5W5QVDI7PVXV67PFJI3ZSVSXHTKPYPSA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.300142221 (469h11m39.171225256s) [noise=true , address=145.79.227.8:4331 , piece_id=ELLVNJFOVY5HSAHZDQYGQVO7OYHFZNFHCBJKHQFV5K6EV2M3JTRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.300506354 (469h11m39.171589389s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.301142795 (469h11m39.17222584s) [noise=true , address=145.79.227.36:4517 , piece_id=HBQU7CMAO2KJS6ODW2WHICBAWQSH6TQQPOQSENKDEISWCWFN2CMA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.301625835 (469h11m39.17270891s) [noise=true , address=145.79.227.45:4229 , piece_id=GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.302170964 (469h11m39.173254009s) [noise=true , address=145.79.227.3:4150 , piece_id=OHGA4EA33AVEDCJLB6QSUJLD3W2TV6YNN6FBVFFKTTSF2B6XDPRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.302629947 (469h11m39.173712982s) [noise=true , address=145.79.227.19:4079 , piece_id=ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303079704 (469h11m39.174162749s) [noise=true , address=145.79.227.15:4875 , piece_id=FUMLT2ODHUE3ETSNWVKBYLSJXYVCHUW5KAJJE5AECNL3IAQYYCLQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303499804 (469h11m39.174582849s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303921386 (469h11m39.175004432s) [noise=true , address=145.79.227.47:4265 , piece_id=TVUXNR6K35A3CWFDN4JRKM4Q6RGBC4LP2XKPXJFN2AJL32ESEBPA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:41.172453460 (469h11m41.043536495s) [upload=411834]: Next returned nil segment Dec 25 04:40:41.172473798 (469h11m41.043556833s) [upload=411834]: Waiting for error group managing segments... Dec 25 04:40:41.483361556 (469h11m41.354444601s) [noise=true , address=145.79.227.37:4567 , piece_id=TZ4C57ISUI3CB3VOLTSM5W5QVDI7PVXV67PFJI3ZSVSXHTKPYPSA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.484283170 (469h11m41.355366205s) [noise=true , address=145.79.227.5:4221 , piece_id=C4SXUVTG2TZ4EIRBGL4TLOLBMYOAZ4KLIQSZOXKEOWAJBQQJLWSA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.515393699 (469h11m41.386476744s) [noise=true , address=145.79.227.51:4411 , piece_id=WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.526179387 (469h11m41.397262422s) [noise=true , address=45.91.116.5:4300 , piece_id=NTWC3RWYQQBBKPMXGUG44R2B5GWC2Z4LDFU3GTWOWCT3JIRIZOOQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.642380892 (469h11m41.513463937s) [noise=true , address=145.79.227.36:4517 , piece_id=HBQU7CMAO2KJS6ODW2WHICBAWQSH6TQQPOQSENKDEISWCWFN2CMA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.662326662 (469h11m41.533409707s) [noise=true , address=145.79.227.3:4150 , piece_id=OHGA4EA33AVEDCJLB6QSUJLD3W2TV6YNN6FBVFFKTTSF2B6XDPRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.758852130 (469h11m41.629935175s) [noise=true , address=145.79.227.47:4265 , piece_id=TVUXNR6K35A3CWFDN4JRKM4Q6RGBC4LP2XKPXJFN2AJL32ESEBPA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.823449082 (469h11m41.694532117s) [noise=true , address=145.79.227.17:4970 , piece_id=6DPKJ762IUN4A7UEZXLDERCVDW24QEKGEHFETAYZOKWQPVN2QJYA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.952636616 (469h11m41.823719661s) [noise=true , address=145.79.227.19:4079 , piece_id=ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.025177123 (469h11m41.896260168s) [noise=true , address=145.79.227.45:4229 , piece_id=GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.222526274 (469h11m42.093609319s) [noise=true , address=145.79.227.8:4331 , piece_id=ELLVNJFOVY5HSAHZDQYGQVO7OYHFZNFHCBJKHQFV5K6EV2M3JTRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.250024374 (469h11m42.121107409s) [noise=true , address=145.79.227.2:4114 , piece_id=5ISXZKQSH4CLLTF3ILGAXHV6STMLRUSZP6UATU2677X2FUPLBCRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.651787921 (469h11m42.522870966s) [noise=true , address=145.79.227.15:4875 , piece_id=FUMLT2ODHUE3ETSNWVKBYLSJXYVCHUW5KAJJE5AECNL3IAQYYCLQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.651814982 (469h11m42.522898027s) [seg_pos={0 0}, upload=411834]: Segment reached optimal threshold of 13 pieces. Dec 25 04:40:42.652052093 (469h11m42.523135129s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: write tcp 188.166.225.170:36650->77.90.31.15:4697: use of closed network connection; storage node overloaded, request limit: 2000 Dec 25 04:40:42.652069537 (469h11m42.523152572s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652177673 (469h11m42.523260708s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd): protocol: write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection; write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection Dec 25 04:40:42.652184456 (469h11m42.523267501s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652230243 (469h11m42.523313308s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq): protocol: write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection Dec 25 04:40:42.652240693 (469h11m42.523323778s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652268265 (469h11m42.52335131s) [seg_pos={0 0}, upload=411834]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 04:40:42.652354770 (469h11m42.523437815s) [upload=411834]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 04:40:42.652361693 (469h11m42.523444738s) [upload=411834]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 04:40:42.959704440 (469h11m42.830787485s) [upload=411834]: Done uploading segments... Dec 25 04:40:42.959728234 (469h11m42.83081127s) [upload=411834]: Upload finished. err: <nil> Duration: 6.0761972s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 5.824963491s trace: https://jaeger.dev.storj.io/trace/681cf842d2260d41 FAIL (2025-12-25 05:04:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, Piece ID: ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ) context deadline exceeded; piecestore: (Node ID: 1DT68m6J4PpDk3XUDhDaaqV46NLKVz4A23716JLFxEAh122GeR, Piece ID: WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA) context deadline exceeded; piecestore: (Node ID: 12wWYnhWHSqqAFgje4p9Cr7WYaJ6dJWsCqvrx8ZqVFoakjuFeka, Piece ID: GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q) context deadline exceeded ======LOGS====== nline: false Dec 25 04:40:39.039085560 (469h11m38.910168605s) [upload=411834]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 04:40:39.039095028 (469h11m38.910178073s) [upload=411834]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 04:40:39.297407613 (469h11m39.168490658s) [seg_pos={0 0}, upload=411834]: Begin upload segment... Dec 25 04:40:39.297471084 (469h11m39.168554119s) [seg_pos={0 0}, upload=411834]: Done begin upload segment. Dec 25 04:40:39.297511581 (469h11m39.168594616s) [noise=true , address=145.79.227.51:4411 , piece_id=WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.297663310 (469h11m39.168746345s) [noise=true , address=145.79.227.17:4970 , piece_id=6DPKJ762IUN4A7UEZXLDERCVDW24QEKGEHFETAYZOKWQPVN2QJYA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.298405574 (469h11m39.169488619s) [noise=true , address=145.79.227.2:4114 , piece_id=5ISXZKQSH4CLLTF3ILGAXHV6STMLRUSZP6UATU2677X2FUPLBCRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.298797961 (469h11m39.169881006s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299124392 (469h11m39.170207437s) [noise=true , address=45.91.116.5:4300 , piece_id=NTWC3RWYQQBBKPMXGUG44R2B5GWC2Z4LDFU3GTWOWCT3JIRIZOOQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299438650 (469h11m39.170521685s) [noise=true , address=145.79.227.5:4221 , piece_id=C4SXUVTG2TZ4EIRBGL4TLOLBMYOAZ4KLIQSZOXKEOWAJBQQJLWSA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299794709 (469h11m39.170877744s) [noise=true , address=145.79.227.37:4567 , piece_id=TZ4C57ISUI3CB3VOLTSM5W5QVDI7PVXV67PFJI3ZSVSXHTKPYPSA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.300142221 (469h11m39.171225256s) [noise=true , address=145.79.227.8:4331 , piece_id=ELLVNJFOVY5HSAHZDQYGQVO7OYHFZNFHCBJKHQFV5K6EV2M3JTRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.300506354 (469h11m39.171589389s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.301142795 (469h11m39.17222584s) [noise=true , address=145.79.227.36:4517 , piece_id=HBQU7CMAO2KJS6ODW2WHICBAWQSH6TQQPOQSENKDEISWCWFN2CMA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.301625835 (469h11m39.17270891s) [noise=true , address=145.79.227.45:4229 , piece_id=GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.302170964 (469h11m39.173254009s) [noise=true , address=145.79.227.3:4150 , piece_id=OHGA4EA33AVEDCJLB6QSUJLD3W2TV6YNN6FBVFFKTTSF2B6XDPRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.302629947 (469h11m39.173712982s) [noise=true , address=145.79.227.19:4079 , piece_id=ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303079704 (469h11m39.174162749s) [noise=true , address=145.79.227.15:4875 , piece_id=FUMLT2ODHUE3ETSNWVKBYLSJXYVCHUW5KAJJE5AECNL3IAQYYCLQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303499804 (469h11m39.174582849s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303921386 (469h11m39.175004432s) [noise=true , address=145.79.227.47:4265 , piece_id=TVUXNR6K35A3CWFDN4JRKM4Q6RGBC4LP2XKPXJFN2AJL32ESEBPA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:41.172453460 (469h11m41.043536495s) [upload=411834]: Next returned nil segment Dec 25 04:40:41.172473798 (469h11m41.043556833s) [upload=411834]: Waiting for error group managing segments... Dec 25 04:40:41.483361556 (469h11m41.354444601s) [noise=true , address=145.79.227.37:4567 , piece_id=TZ4C57ISUI3CB3VOLTSM5W5QVDI7PVXV67PFJI3ZSVSXHTKPYPSA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.484283170 (469h11m41.355366205s) [noise=true , address=145.79.227.5:4221 , piece_id=C4SXUVTG2TZ4EIRBGL4TLOLBMYOAZ4KLIQSZOXKEOWAJBQQJLWSA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.515393699 (469h11m41.386476744s) [noise=true , address=145.79.227.51:4411 , piece_id=WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.526179387 (469h11m41.397262422s) [noise=true , address=45.91.116.5:4300 , piece_id=NTWC3RWYQQBBKPMXGUG44R2B5GWC2Z4LDFU3GTWOWCT3JIRIZOOQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.642380892 (469h11m41.513463937s) [noise=true , address=145.79.227.36:4517 , piece_id=HBQU7CMAO2KJS6ODW2WHICBAWQSH6TQQPOQSENKDEISWCWFN2CMA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.662326662 (469h11m41.533409707s) [noise=true , address=145.79.227.3:4150 , piece_id=OHGA4EA33AVEDCJLB6QSUJLD3W2TV6YNN6FBVFFKTTSF2B6XDPRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.758852130 (469h11m41.629935175s) [noise=true , address=145.79.227.47:4265 , piece_id=TVUXNR6K35A3CWFDN4JRKM4Q6RGBC4LP2XKPXJFN2AJL32ESEBPA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.823449082 (469h11m41.694532117s) [noise=true , address=145.79.227.17:4970 , piece_id=6DPKJ762IUN4A7UEZXLDERCVDW24QEKGEHFETAYZOKWQPVN2QJYA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.952636616 (469h11m41.823719661s) [noise=true , address=145.79.227.19:4079 , piece_id=ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.025177123 (469h11m41.896260168s) [noise=true , address=145.79.227.45:4229 , piece_id=GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.222526274 (469h11m42.093609319s) [noise=true , address=145.79.227.8:4331 , piece_id=ELLVNJFOVY5HSAHZDQYGQVO7OYHFZNFHCBJKHQFV5K6EV2M3JTRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.250024374 (469h11m42.121107409s) [noise=true , address=145.79.227.2:4114 , piece_id=5ISXZKQSH4CLLTF3ILGAXHV6STMLRUSZP6UATU2677X2FUPLBCRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.651787921 (469h11m42.522870966s) [noise=true , address=145.79.227.15:4875 , piece_id=FUMLT2ODHUE3ETSNWVKBYLSJXYVCHUW5KAJJE5AECNL3IAQYYCLQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.651814982 (469h11m42.522898027s) [seg_pos={0 0}, upload=411834]: Segment reached optimal threshold of 13 pieces. Dec 25 04:40:42.652052093 (469h11m42.523135129s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: write tcp 188.166.225.170:36650->77.90.31.15:4697: use of closed network connection; storage node overloaded, request limit: 2000 Dec 25 04:40:42.652069537 (469h11m42.523152572s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652177673 (469h11m42.523260708s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd): protocol: write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection; write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection Dec 25 04:40:42.652184456 (469h11m42.523267501s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652230243 (469h11m42.523313308s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq): protocol: write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection Dec 25 04:40:42.652240693 (469h11m42.523323778s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652268265 (469h11m42.52335131s) [seg_pos={0 0}, upload=411834]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 04:40:42.652354770 (469h11m42.523437815s) [upload=411834]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 04:40:42.652361693 (469h11m42.523444738s) [upload=411834]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 04:40:42.959704440 (469h11m42.830787485s) [upload=411834]: Done uploading segments... Dec 25 04:40:42.959728234 (469h11m42.83081127s) [upload=411834]: Upload finished. err: <nil> Duration: 5.825004078s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 5.74011557s trace: https://jaeger.dev.storj.io/trace/681cf842d2260d41 FAIL (2025-12-25 05:04:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, Piece ID: ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ) context deadline exceeded; piecestore: (Node ID: 1DT68m6J4PpDk3XUDhDaaqV46NLKVz4A23716JLFxEAh122GeR, Piece ID: WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA) context deadline exceeded; piecestore: (Node ID: 12wWYnhWHSqqAFgje4p9Cr7WYaJ6dJWsCqvrx8ZqVFoakjuFeka, Piece ID: GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q) context deadline exceeded ======LOGS====== nline: false Dec 25 04:40:39.039085560 (469h11m38.910168605s) [upload=411834]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 04:40:39.039095028 (469h11m38.910178073s) [upload=411834]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 04:40:39.297407613 (469h11m39.168490658s) [seg_pos={0 0}, upload=411834]: Begin upload segment... Dec 25 04:40:39.297471084 (469h11m39.168554119s) [seg_pos={0 0}, upload=411834]: Done begin upload segment. Dec 25 04:40:39.297511581 (469h11m39.168594616s) [noise=true , address=145.79.227.51:4411 , piece_id=WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.297663310 (469h11m39.168746345s) [noise=true , address=145.79.227.17:4970 , piece_id=6DPKJ762IUN4A7UEZXLDERCVDW24QEKGEHFETAYZOKWQPVN2QJYA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.298405574 (469h11m39.169488619s) [noise=true , address=145.79.227.2:4114 , piece_id=5ISXZKQSH4CLLTF3ILGAXHV6STMLRUSZP6UATU2677X2FUPLBCRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.298797961 (469h11m39.169881006s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299124392 (469h11m39.170207437s) [noise=true , address=45.91.116.5:4300 , piece_id=NTWC3RWYQQBBKPMXGUG44R2B5GWC2Z4LDFU3GTWOWCT3JIRIZOOQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299438650 (469h11m39.170521685s) [noise=true , address=145.79.227.5:4221 , piece_id=C4SXUVTG2TZ4EIRBGL4TLOLBMYOAZ4KLIQSZOXKEOWAJBQQJLWSA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299794709 (469h11m39.170877744s) [noise=true , address=145.79.227.37:4567 , piece_id=TZ4C57ISUI3CB3VOLTSM5W5QVDI7PVXV67PFJI3ZSVSXHTKPYPSA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.300142221 (469h11m39.171225256s) [noise=true , address=145.79.227.8:4331 , piece_id=ELLVNJFOVY5HSAHZDQYGQVO7OYHFZNFHCBJKHQFV5K6EV2M3JTRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.300506354 (469h11m39.171589389s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.301142795 (469h11m39.17222584s) [noise=true , address=145.79.227.36:4517 , piece_id=HBQU7CMAO2KJS6ODW2WHICBAWQSH6TQQPOQSENKDEISWCWFN2CMA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.301625835 (469h11m39.17270891s) [noise=true , address=145.79.227.45:4229 , piece_id=GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.302170964 (469h11m39.173254009s) [noise=true , address=145.79.227.3:4150 , piece_id=OHGA4EA33AVEDCJLB6QSUJLD3W2TV6YNN6FBVFFKTTSF2B6XDPRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.302629947 (469h11m39.173712982s) [noise=true , address=145.79.227.19:4079 , piece_id=ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303079704 (469h11m39.174162749s) [noise=true , address=145.79.227.15:4875 , piece_id=FUMLT2ODHUE3ETSNWVKBYLSJXYVCHUW5KAJJE5AECNL3IAQYYCLQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303499804 (469h11m39.174582849s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303921386 (469h11m39.175004432s) [noise=true , address=145.79.227.47:4265 , piece_id=TVUXNR6K35A3CWFDN4JRKM4Q6RGBC4LP2XKPXJFN2AJL32ESEBPA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:41.172453460 (469h11m41.043536495s) [upload=411834]: Next returned nil segment Dec 25 04:40:41.172473798 (469h11m41.043556833s) [upload=411834]: Waiting for error group managing segments... Dec 25 04:40:41.483361556 (469h11m41.354444601s) [noise=true , address=145.79.227.37:4567 , piece_id=TZ4C57ISUI3CB3VOLTSM5W5QVDI7PVXV67PFJI3ZSVSXHTKPYPSA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.484283170 (469h11m41.355366205s) [noise=true , address=145.79.227.5:4221 , piece_id=C4SXUVTG2TZ4EIRBGL4TLOLBMYOAZ4KLIQSZOXKEOWAJBQQJLWSA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.515393699 (469h11m41.386476744s) [noise=true , address=145.79.227.51:4411 , piece_id=WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.526179387 (469h11m41.397262422s) [noise=true , address=45.91.116.5:4300 , piece_id=NTWC3RWYQQBBKPMXGUG44R2B5GWC2Z4LDFU3GTWOWCT3JIRIZOOQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.642380892 (469h11m41.513463937s) [noise=true , address=145.79.227.36:4517 , piece_id=HBQU7CMAO2KJS6ODW2WHICBAWQSH6TQQPOQSENKDEISWCWFN2CMA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.662326662 (469h11m41.533409707s) [noise=true , address=145.79.227.3:4150 , piece_id=OHGA4EA33AVEDCJLB6QSUJLD3W2TV6YNN6FBVFFKTTSF2B6XDPRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.758852130 (469h11m41.629935175s) [noise=true , address=145.79.227.47:4265 , piece_id=TVUXNR6K35A3CWFDN4JRKM4Q6RGBC4LP2XKPXJFN2AJL32ESEBPA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.823449082 (469h11m41.694532117s) [noise=true , address=145.79.227.17:4970 , piece_id=6DPKJ762IUN4A7UEZXLDERCVDW24QEKGEHFETAYZOKWQPVN2QJYA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.952636616 (469h11m41.823719661s) [noise=true , address=145.79.227.19:4079 , piece_id=ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.025177123 (469h11m41.896260168s) [noise=true , address=145.79.227.45:4229 , piece_id=GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.222526274 (469h11m42.093609319s) [noise=true , address=145.79.227.8:4331 , piece_id=ELLVNJFOVY5HSAHZDQYGQVO7OYHFZNFHCBJKHQFV5K6EV2M3JTRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.250024374 (469h11m42.121107409s) [noise=true , address=145.79.227.2:4114 , piece_id=5ISXZKQSH4CLLTF3ILGAXHV6STMLRUSZP6UATU2677X2FUPLBCRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.651787921 (469h11m42.522870966s) [noise=true , address=145.79.227.15:4875 , piece_id=FUMLT2ODHUE3ETSNWVKBYLSJXYVCHUW5KAJJE5AECNL3IAQYYCLQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.651814982 (469h11m42.522898027s) [seg_pos={0 0}, upload=411834]: Segment reached optimal threshold of 13 pieces. Dec 25 04:40:42.652052093 (469h11m42.523135129s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: write tcp 188.166.225.170:36650->77.90.31.15:4697: use of closed network connection; storage node overloaded, request limit: 2000 Dec 25 04:40:42.652069537 (469h11m42.523152572s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652177673 (469h11m42.523260708s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd): protocol: write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection; write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection Dec 25 04:40:42.652184456 (469h11m42.523267501s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652230243 (469h11m42.523313308s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq): protocol: write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection Dec 25 04:40:42.652240693 (469h11m42.523323778s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652268265 (469h11m42.52335131s) [seg_pos={0 0}, upload=411834]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 04:40:42.652354770 (469h11m42.523437815s) [upload=411834]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 04:40:42.652361693 (469h11m42.523444738s) [upload=411834]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 04:40:42.959704440 (469h11m42.830787485s) [upload=411834]: Done uploading segments... Dec 25 04:40:42.959728234 (469h11m42.83081127s) [upload=411834]: Upload finished. err: <nil> Duration: 5.740157068s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 5.913805437s trace: https://jaeger.dev.storj.io/trace/681cf842d2260d41 FAIL (2025-12-25 05:03:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, Piece ID: ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ) context deadline exceeded; piecestore: (Node ID: 1DT68m6J4PpDk3XUDhDaaqV46NLKVz4A23716JLFxEAh122GeR, Piece ID: WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA) context deadline exceeded; piecestore: (Node ID: 12wWYnhWHSqqAFgje4p9Cr7WYaJ6dJWsCqvrx8ZqVFoakjuFeka, Piece ID: GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q) context deadline exceeded ======LOGS====== nline: false Dec 25 04:40:39.039085560 (469h11m38.910168605s) [upload=411834]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 04:40:39.039095028 (469h11m38.910178073s) [upload=411834]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 04:40:39.297407613 (469h11m39.168490658s) [seg_pos={0 0}, upload=411834]: Begin upload segment... Dec 25 04:40:39.297471084 (469h11m39.168554119s) [seg_pos={0 0}, upload=411834]: Done begin upload segment. Dec 25 04:40:39.297511581 (469h11m39.168594616s) [noise=true , address=145.79.227.51:4411 , piece_id=WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.297663310 (469h11m39.168746345s) [noise=true , address=145.79.227.17:4970 , piece_id=6DPKJ762IUN4A7UEZXLDERCVDW24QEKGEHFETAYZOKWQPVN2QJYA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.298405574 (469h11m39.169488619s) [noise=true , address=145.79.227.2:4114 , piece_id=5ISXZKQSH4CLLTF3ILGAXHV6STMLRUSZP6UATU2677X2FUPLBCRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.298797961 (469h11m39.169881006s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299124392 (469h11m39.170207437s) [noise=true , address=45.91.116.5:4300 , piece_id=NTWC3RWYQQBBKPMXGUG44R2B5GWC2Z4LDFU3GTWOWCT3JIRIZOOQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299438650 (469h11m39.170521685s) [noise=true , address=145.79.227.5:4221 , piece_id=C4SXUVTG2TZ4EIRBGL4TLOLBMYOAZ4KLIQSZOXKEOWAJBQQJLWSA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299794709 (469h11m39.170877744s) [noise=true , address=145.79.227.37:4567 , piece_id=TZ4C57ISUI3CB3VOLTSM5W5QVDI7PVXV67PFJI3ZSVSXHTKPYPSA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.300142221 (469h11m39.171225256s) [noise=true , address=145.79.227.8:4331 , piece_id=ELLVNJFOVY5HSAHZDQYGQVO7OYHFZNFHCBJKHQFV5K6EV2M3JTRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.300506354 (469h11m39.171589389s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.301142795 (469h11m39.17222584s) [noise=true , address=145.79.227.36:4517 , piece_id=HBQU7CMAO2KJS6ODW2WHICBAWQSH6TQQPOQSENKDEISWCWFN2CMA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.301625835 (469h11m39.17270891s) [noise=true , address=145.79.227.45:4229 , piece_id=GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.302170964 (469h11m39.173254009s) [noise=true , address=145.79.227.3:4150 , piece_id=OHGA4EA33AVEDCJLB6QSUJLD3W2TV6YNN6FBVFFKTTSF2B6XDPRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.302629947 (469h11m39.173712982s) [noise=true , address=145.79.227.19:4079 , piece_id=ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303079704 (469h11m39.174162749s) [noise=true , address=145.79.227.15:4875 , piece_id=FUMLT2ODHUE3ETSNWVKBYLSJXYVCHUW5KAJJE5AECNL3IAQYYCLQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303499804 (469h11m39.174582849s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303921386 (469h11m39.175004432s) [noise=true , address=145.79.227.47:4265 , piece_id=TVUXNR6K35A3CWFDN4JRKM4Q6RGBC4LP2XKPXJFN2AJL32ESEBPA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:41.172453460 (469h11m41.043536495s) [upload=411834]: Next returned nil segment Dec 25 04:40:41.172473798 (469h11m41.043556833s) [upload=411834]: Waiting for error group managing segments... Dec 25 04:40:41.483361556 (469h11m41.354444601s) [noise=true , address=145.79.227.37:4567 , piece_id=TZ4C57ISUI3CB3VOLTSM5W5QVDI7PVXV67PFJI3ZSVSXHTKPYPSA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.484283170 (469h11m41.355366205s) [noise=true , address=145.79.227.5:4221 , piece_id=C4SXUVTG2TZ4EIRBGL4TLOLBMYOAZ4KLIQSZOXKEOWAJBQQJLWSA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.515393699 (469h11m41.386476744s) [noise=true , address=145.79.227.51:4411 , piece_id=WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.526179387 (469h11m41.397262422s) [noise=true , address=45.91.116.5:4300 , piece_id=NTWC3RWYQQBBKPMXGUG44R2B5GWC2Z4LDFU3GTWOWCT3JIRIZOOQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.642380892 (469h11m41.513463937s) [noise=true , address=145.79.227.36:4517 , piece_id=HBQU7CMAO2KJS6ODW2WHICBAWQSH6TQQPOQSENKDEISWCWFN2CMA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.662326662 (469h11m41.533409707s) [noise=true , address=145.79.227.3:4150 , piece_id=OHGA4EA33AVEDCJLB6QSUJLD3W2TV6YNN6FBVFFKTTSF2B6XDPRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.758852130 (469h11m41.629935175s) [noise=true , address=145.79.227.47:4265 , piece_id=TVUXNR6K35A3CWFDN4JRKM4Q6RGBC4LP2XKPXJFN2AJL32ESEBPA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.823449082 (469h11m41.694532117s) [noise=true , address=145.79.227.17:4970 , piece_id=6DPKJ762IUN4A7UEZXLDERCVDW24QEKGEHFETAYZOKWQPVN2QJYA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.952636616 (469h11m41.823719661s) [noise=true , address=145.79.227.19:4079 , piece_id=ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.025177123 (469h11m41.896260168s) [noise=true , address=145.79.227.45:4229 , piece_id=GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.222526274 (469h11m42.093609319s) [noise=true , address=145.79.227.8:4331 , piece_id=ELLVNJFOVY5HSAHZDQYGQVO7OYHFZNFHCBJKHQFV5K6EV2M3JTRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.250024374 (469h11m42.121107409s) [noise=true , address=145.79.227.2:4114 , piece_id=5ISXZKQSH4CLLTF3ILGAXHV6STMLRUSZP6UATU2677X2FUPLBCRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.651787921 (469h11m42.522870966s) [noise=true , address=145.79.227.15:4875 , piece_id=FUMLT2ODHUE3ETSNWVKBYLSJXYVCHUW5KAJJE5AECNL3IAQYYCLQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.651814982 (469h11m42.522898027s) [seg_pos={0 0}, upload=411834]: Segment reached optimal threshold of 13 pieces. Dec 25 04:40:42.652052093 (469h11m42.523135129s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: write tcp 188.166.225.170:36650->77.90.31.15:4697: use of closed network connection; storage node overloaded, request limit: 2000 Dec 25 04:40:42.652069537 (469h11m42.523152572s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652177673 (469h11m42.523260708s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd): protocol: write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection; write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection Dec 25 04:40:42.652184456 (469h11m42.523267501s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652230243 (469h11m42.523313308s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq): protocol: write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection Dec 25 04:40:42.652240693 (469h11m42.523323778s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652268265 (469h11m42.52335131s) [seg_pos={0 0}, upload=411834]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 04:40:42.652354770 (469h11m42.523437815s) [upload=411834]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 04:40:42.652361693 (469h11m42.523444738s) [upload=411834]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 04:40:42.959704440 (469h11m42.830787485s) [upload=411834]: Done uploading segments... Dec 25 04:40:42.959728234 (469h11m42.83081127s) [upload=411834]: Upload finished. err: <nil> Duration: 5.913834193s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 5.808622438s trace: https://jaeger.dev.storj.io/trace/681cf842d2260d41 FAIL (2025-12-25 05:02:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, Piece ID: ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ) context deadline exceeded; piecestore: (Node ID: 1DT68m6J4PpDk3XUDhDaaqV46NLKVz4A23716JLFxEAh122GeR, Piece ID: WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA) context deadline exceeded; piecestore: (Node ID: 12wWYnhWHSqqAFgje4p9Cr7WYaJ6dJWsCqvrx8ZqVFoakjuFeka, Piece ID: GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q) context deadline exceeded ======LOGS====== nline: false Dec 25 04:40:39.039085560 (469h11m38.910168605s) [upload=411834]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 04:40:39.039095028 (469h11m38.910178073s) [upload=411834]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 04:40:39.297407613 (469h11m39.168490658s) [seg_pos={0 0}, upload=411834]: Begin upload segment... Dec 25 04:40:39.297471084 (469h11m39.168554119s) [seg_pos={0 0}, upload=411834]: Done begin upload segment. Dec 25 04:40:39.297511581 (469h11m39.168594616s) [noise=true , address=145.79.227.51:4411 , piece_id=WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.297663310 (469h11m39.168746345s) [noise=true , address=145.79.227.17:4970 , piece_id=6DPKJ762IUN4A7UEZXLDERCVDW24QEKGEHFETAYZOKWQPVN2QJYA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.298405574 (469h11m39.169488619s) [noise=true , address=145.79.227.2:4114 , piece_id=5ISXZKQSH4CLLTF3ILGAXHV6STMLRUSZP6UATU2677X2FUPLBCRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.298797961 (469h11m39.169881006s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299124392 (469h11m39.170207437s) [noise=true , address=45.91.116.5:4300 , piece_id=NTWC3RWYQQBBKPMXGUG44R2B5GWC2Z4LDFU3GTWOWCT3JIRIZOOQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299438650 (469h11m39.170521685s) [noise=true , address=145.79.227.5:4221 , piece_id=C4SXUVTG2TZ4EIRBGL4TLOLBMYOAZ4KLIQSZOXKEOWAJBQQJLWSA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299794709 (469h11m39.170877744s) [noise=true , address=145.79.227.37:4567 , piece_id=TZ4C57ISUI3CB3VOLTSM5W5QVDI7PVXV67PFJI3ZSVSXHTKPYPSA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.300142221 (469h11m39.171225256s) [noise=true , address=145.79.227.8:4331 , piece_id=ELLVNJFOVY5HSAHZDQYGQVO7OYHFZNFHCBJKHQFV5K6EV2M3JTRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.300506354 (469h11m39.171589389s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.301142795 (469h11m39.17222584s) [noise=true , address=145.79.227.36:4517 , piece_id=HBQU7CMAO2KJS6ODW2WHICBAWQSH6TQQPOQSENKDEISWCWFN2CMA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.301625835 (469h11m39.17270891s) [noise=true , address=145.79.227.45:4229 , piece_id=GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.302170964 (469h11m39.173254009s) [noise=true , address=145.79.227.3:4150 , piece_id=OHGA4EA33AVEDCJLB6QSUJLD3W2TV6YNN6FBVFFKTTSF2B6XDPRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.302629947 (469h11m39.173712982s) [noise=true , address=145.79.227.19:4079 , piece_id=ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303079704 (469h11m39.174162749s) [noise=true , address=145.79.227.15:4875 , piece_id=FUMLT2ODHUE3ETSNWVKBYLSJXYVCHUW5KAJJE5AECNL3IAQYYCLQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303499804 (469h11m39.174582849s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303921386 (469h11m39.175004432s) [noise=true , address=145.79.227.47:4265 , piece_id=TVUXNR6K35A3CWFDN4JRKM4Q6RGBC4LP2XKPXJFN2AJL32ESEBPA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:41.172453460 (469h11m41.043536495s) [upload=411834]: Next returned nil segment Dec 25 04:40:41.172473798 (469h11m41.043556833s) [upload=411834]: Waiting for error group managing segments... Dec 25 04:40:41.483361556 (469h11m41.354444601s) [noise=true , address=145.79.227.37:4567 , piece_id=TZ4C57ISUI3CB3VOLTSM5W5QVDI7PVXV67PFJI3ZSVSXHTKPYPSA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.484283170 (469h11m41.355366205s) [noise=true , address=145.79.227.5:4221 , piece_id=C4SXUVTG2TZ4EIRBGL4TLOLBMYOAZ4KLIQSZOXKEOWAJBQQJLWSA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.515393699 (469h11m41.386476744s) [noise=true , address=145.79.227.51:4411 , piece_id=WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.526179387 (469h11m41.397262422s) [noise=true , address=45.91.116.5:4300 , piece_id=NTWC3RWYQQBBKPMXGUG44R2B5GWC2Z4LDFU3GTWOWCT3JIRIZOOQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.642380892 (469h11m41.513463937s) [noise=true , address=145.79.227.36:4517 , piece_id=HBQU7CMAO2KJS6ODW2WHICBAWQSH6TQQPOQSENKDEISWCWFN2CMA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.662326662 (469h11m41.533409707s) [noise=true , address=145.79.227.3:4150 , piece_id=OHGA4EA33AVEDCJLB6QSUJLD3W2TV6YNN6FBVFFKTTSF2B6XDPRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.758852130 (469h11m41.629935175s) [noise=true , address=145.79.227.47:4265 , piece_id=TVUXNR6K35A3CWFDN4JRKM4Q6RGBC4LP2XKPXJFN2AJL32ESEBPA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.823449082 (469h11m41.694532117s) [noise=true , address=145.79.227.17:4970 , piece_id=6DPKJ762IUN4A7UEZXLDERCVDW24QEKGEHFETAYZOKWQPVN2QJYA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.952636616 (469h11m41.823719661s) [noise=true , address=145.79.227.19:4079 , piece_id=ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.025177123 (469h11m41.896260168s) [noise=true , address=145.79.227.45:4229 , piece_id=GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.222526274 (469h11m42.093609319s) [noise=true , address=145.79.227.8:4331 , piece_id=ELLVNJFOVY5HSAHZDQYGQVO7OYHFZNFHCBJKHQFV5K6EV2M3JTRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.250024374 (469h11m42.121107409s) [noise=true , address=145.79.227.2:4114 , piece_id=5ISXZKQSH4CLLTF3ILGAXHV6STMLRUSZP6UATU2677X2FUPLBCRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.651787921 (469h11m42.522870966s) [noise=true , address=145.79.227.15:4875 , piece_id=FUMLT2ODHUE3ETSNWVKBYLSJXYVCHUW5KAJJE5AECNL3IAQYYCLQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.651814982 (469h11m42.522898027s) [seg_pos={0 0}, upload=411834]: Segment reached optimal threshold of 13 pieces. Dec 25 04:40:42.652052093 (469h11m42.523135129s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: write tcp 188.166.225.170:36650->77.90.31.15:4697: use of closed network connection; storage node overloaded, request limit: 2000 Dec 25 04:40:42.652069537 (469h11m42.523152572s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652177673 (469h11m42.523260708s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd): protocol: write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection; write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection Dec 25 04:40:42.652184456 (469h11m42.523267501s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652230243 (469h11m42.523313308s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq): protocol: write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection Dec 25 04:40:42.652240693 (469h11m42.523323778s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652268265 (469h11m42.52335131s) [seg_pos={0 0}, upload=411834]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 04:40:42.652354770 (469h11m42.523437815s) [upload=411834]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 04:40:42.652361693 (469h11m42.523444738s) [upload=411834]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 04:40:42.959704440 (469h11m42.830787485s) [upload=411834]: Done uploading segments... Dec 25 04:40:42.959728234 (469h11m42.83081127s) [upload=411834]: Upload finished. err: <nil> Duration: 5.808651184s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 6.01305729s trace: https://jaeger.dev.storj.io/trace/681cf842d2260d41 FAIL (2025-12-25 05:01:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, Piece ID: ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ) context deadline exceeded; piecestore: (Node ID: 1DT68m6J4PpDk3XUDhDaaqV46NLKVz4A23716JLFxEAh122GeR, Piece ID: WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA) context deadline exceeded; piecestore: (Node ID: 12wWYnhWHSqqAFgje4p9Cr7WYaJ6dJWsCqvrx8ZqVFoakjuFeka, Piece ID: GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q) context deadline exceeded ======LOGS====== nline: false Dec 25 04:40:39.039085560 (469h11m38.910168605s) [upload=411834]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 04:40:39.039095028 (469h11m38.910178073s) [upload=411834]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 04:40:39.297407613 (469h11m39.168490658s) [seg_pos={0 0}, upload=411834]: Begin upload segment... Dec 25 04:40:39.297471084 (469h11m39.168554119s) [seg_pos={0 0}, upload=411834]: Done begin upload segment. Dec 25 04:40:39.297511581 (469h11m39.168594616s) [noise=true , address=145.79.227.51:4411 , piece_id=WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.297663310 (469h11m39.168746345s) [noise=true , address=145.79.227.17:4970 , piece_id=6DPKJ762IUN4A7UEZXLDERCVDW24QEKGEHFETAYZOKWQPVN2QJYA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.298405574 (469h11m39.169488619s) [noise=true , address=145.79.227.2:4114 , piece_id=5ISXZKQSH4CLLTF3ILGAXHV6STMLRUSZP6UATU2677X2FUPLBCRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.298797961 (469h11m39.169881006s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299124392 (469h11m39.170207437s) [noise=true , address=45.91.116.5:4300 , piece_id=NTWC3RWYQQBBKPMXGUG44R2B5GWC2Z4LDFU3GTWOWCT3JIRIZOOQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299438650 (469h11m39.170521685s) [noise=true , address=145.79.227.5:4221 , piece_id=C4SXUVTG2TZ4EIRBGL4TLOLBMYOAZ4KLIQSZOXKEOWAJBQQJLWSA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299794709 (469h11m39.170877744s) [noise=true , address=145.79.227.37:4567 , piece_id=TZ4C57ISUI3CB3VOLTSM5W5QVDI7PVXV67PFJI3ZSVSXHTKPYPSA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.300142221 (469h11m39.171225256s) [noise=true , address=145.79.227.8:4331 , piece_id=ELLVNJFOVY5HSAHZDQYGQVO7OYHFZNFHCBJKHQFV5K6EV2M3JTRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.300506354 (469h11m39.171589389s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.301142795 (469h11m39.17222584s) [noise=true , address=145.79.227.36:4517 , piece_id=HBQU7CMAO2KJS6ODW2WHICBAWQSH6TQQPOQSENKDEISWCWFN2CMA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.301625835 (469h11m39.17270891s) [noise=true , address=145.79.227.45:4229 , piece_id=GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.302170964 (469h11m39.173254009s) [noise=true , address=145.79.227.3:4150 , piece_id=OHGA4EA33AVEDCJLB6QSUJLD3W2TV6YNN6FBVFFKTTSF2B6XDPRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.302629947 (469h11m39.173712982s) [noise=true , address=145.79.227.19:4079 , piece_id=ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303079704 (469h11m39.174162749s) [noise=true , address=145.79.227.15:4875 , piece_id=FUMLT2ODHUE3ETSNWVKBYLSJXYVCHUW5KAJJE5AECNL3IAQYYCLQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303499804 (469h11m39.174582849s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303921386 (469h11m39.175004432s) [noise=true , address=145.79.227.47:4265 , piece_id=TVUXNR6K35A3CWFDN4JRKM4Q6RGBC4LP2XKPXJFN2AJL32ESEBPA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:41.172453460 (469h11m41.043536495s) [upload=411834]: Next returned nil segment Dec 25 04:40:41.172473798 (469h11m41.043556833s) [upload=411834]: Waiting for error group managing segments... Dec 25 04:40:41.483361556 (469h11m41.354444601s) [noise=true , address=145.79.227.37:4567 , piece_id=TZ4C57ISUI3CB3VOLTSM5W5QVDI7PVXV67PFJI3ZSVSXHTKPYPSA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.484283170 (469h11m41.355366205s) [noise=true , address=145.79.227.5:4221 , piece_id=C4SXUVTG2TZ4EIRBGL4TLOLBMYOAZ4KLIQSZOXKEOWAJBQQJLWSA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.515393699 (469h11m41.386476744s) [noise=true , address=145.79.227.51:4411 , piece_id=WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.526179387 (469h11m41.397262422s) [noise=true , address=45.91.116.5:4300 , piece_id=NTWC3RWYQQBBKPMXGUG44R2B5GWC2Z4LDFU3GTWOWCT3JIRIZOOQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.642380892 (469h11m41.513463937s) [noise=true , address=145.79.227.36:4517 , piece_id=HBQU7CMAO2KJS6ODW2WHICBAWQSH6TQQPOQSENKDEISWCWFN2CMA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.662326662 (469h11m41.533409707s) [noise=true , address=145.79.227.3:4150 , piece_id=OHGA4EA33AVEDCJLB6QSUJLD3W2TV6YNN6FBVFFKTTSF2B6XDPRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.758852130 (469h11m41.629935175s) [noise=true , address=145.79.227.47:4265 , piece_id=TVUXNR6K35A3CWFDN4JRKM4Q6RGBC4LP2XKPXJFN2AJL32ESEBPA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.823449082 (469h11m41.694532117s) [noise=true , address=145.79.227.17:4970 , piece_id=6DPKJ762IUN4A7UEZXLDERCVDW24QEKGEHFETAYZOKWQPVN2QJYA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.952636616 (469h11m41.823719661s) [noise=true , address=145.79.227.19:4079 , piece_id=ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.025177123 (469h11m41.896260168s) [noise=true , address=145.79.227.45:4229 , piece_id=GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.222526274 (469h11m42.093609319s) [noise=true , address=145.79.227.8:4331 , piece_id=ELLVNJFOVY5HSAHZDQYGQVO7OYHFZNFHCBJKHQFV5K6EV2M3JTRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.250024374 (469h11m42.121107409s) [noise=true , address=145.79.227.2:4114 , piece_id=5ISXZKQSH4CLLTF3ILGAXHV6STMLRUSZP6UATU2677X2FUPLBCRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.651787921 (469h11m42.522870966s) [noise=true , address=145.79.227.15:4875 , piece_id=FUMLT2ODHUE3ETSNWVKBYLSJXYVCHUW5KAJJE5AECNL3IAQYYCLQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.651814982 (469h11m42.522898027s) [seg_pos={0 0}, upload=411834]: Segment reached optimal threshold of 13 pieces. Dec 25 04:40:42.652052093 (469h11m42.523135129s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: write tcp 188.166.225.170:36650->77.90.31.15:4697: use of closed network connection; storage node overloaded, request limit: 2000 Dec 25 04:40:42.652069537 (469h11m42.523152572s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652177673 (469h11m42.523260708s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd): protocol: write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection; write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection Dec 25 04:40:42.652184456 (469h11m42.523267501s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652230243 (469h11m42.523313308s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq): protocol: write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection Dec 25 04:40:42.652240693 (469h11m42.523323778s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652268265 (469h11m42.52335131s) [seg_pos={0 0}, upload=411834]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 04:40:42.652354770 (469h11m42.523437815s) [upload=411834]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 04:40:42.652361693 (469h11m42.523444738s) [upload=411834]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 04:40:42.959704440 (469h11m42.830787485s) [upload=411834]: Done uploading segments... Dec 25 04:40:42.959728234 (469h11m42.83081127s) [upload=411834]: Upload finished. err: <nil> Duration: 6.01309418s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 6.122129603s trace: https://jaeger.dev.storj.io/trace/681cf842d2260d41 FAIL (2025-12-25 05:01:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, Piece ID: ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ) context deadline exceeded; piecestore: (Node ID: 1DT68m6J4PpDk3XUDhDaaqV46NLKVz4A23716JLFxEAh122GeR, Piece ID: WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA) context deadline exceeded; piecestore: (Node ID: 12wWYnhWHSqqAFgje4p9Cr7WYaJ6dJWsCqvrx8ZqVFoakjuFeka, Piece ID: GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q) context deadline exceeded ======LOGS====== nline: false Dec 25 04:40:39.039085560 (469h11m38.910168605s) [upload=411834]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 04:40:39.039095028 (469h11m38.910178073s) [upload=411834]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 04:40:39.297407613 (469h11m39.168490658s) [seg_pos={0 0}, upload=411834]: Begin upload segment... Dec 25 04:40:39.297471084 (469h11m39.168554119s) [seg_pos={0 0}, upload=411834]: Done begin upload segment. Dec 25 04:40:39.297511581 (469h11m39.168594616s) [noise=true , address=145.79.227.51:4411 , piece_id=WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.297663310 (469h11m39.168746345s) [noise=true , address=145.79.227.17:4970 , piece_id=6DPKJ762IUN4A7UEZXLDERCVDW24QEKGEHFETAYZOKWQPVN2QJYA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.298405574 (469h11m39.169488619s) [noise=true , address=145.79.227.2:4114 , piece_id=5ISXZKQSH4CLLTF3ILGAXHV6STMLRUSZP6UATU2677X2FUPLBCRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.298797961 (469h11m39.169881006s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299124392 (469h11m39.170207437s) [noise=true , address=45.91.116.5:4300 , piece_id=NTWC3RWYQQBBKPMXGUG44R2B5GWC2Z4LDFU3GTWOWCT3JIRIZOOQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299438650 (469h11m39.170521685s) [noise=true , address=145.79.227.5:4221 , piece_id=C4SXUVTG2TZ4EIRBGL4TLOLBMYOAZ4KLIQSZOXKEOWAJBQQJLWSA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299794709 (469h11m39.170877744s) [noise=true , address=145.79.227.37:4567 , piece_id=TZ4C57ISUI3CB3VOLTSM5W5QVDI7PVXV67PFJI3ZSVSXHTKPYPSA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.300142221 (469h11m39.171225256s) [noise=true , address=145.79.227.8:4331 , piece_id=ELLVNJFOVY5HSAHZDQYGQVO7OYHFZNFHCBJKHQFV5K6EV2M3JTRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.300506354 (469h11m39.171589389s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.301142795 (469h11m39.17222584s) [noise=true , address=145.79.227.36:4517 , piece_id=HBQU7CMAO2KJS6ODW2WHICBAWQSH6TQQPOQSENKDEISWCWFN2CMA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.301625835 (469h11m39.17270891s) [noise=true , address=145.79.227.45:4229 , piece_id=GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.302170964 (469h11m39.173254009s) [noise=true , address=145.79.227.3:4150 , piece_id=OHGA4EA33AVEDCJLB6QSUJLD3W2TV6YNN6FBVFFKTTSF2B6XDPRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.302629947 (469h11m39.173712982s) [noise=true , address=145.79.227.19:4079 , piece_id=ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303079704 (469h11m39.174162749s) [noise=true , address=145.79.227.15:4875 , piece_id=FUMLT2ODHUE3ETSNWVKBYLSJXYVCHUW5KAJJE5AECNL3IAQYYCLQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303499804 (469h11m39.174582849s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303921386 (469h11m39.175004432s) [noise=true , address=145.79.227.47:4265 , piece_id=TVUXNR6K35A3CWFDN4JRKM4Q6RGBC4LP2XKPXJFN2AJL32ESEBPA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:41.172453460 (469h11m41.043536495s) [upload=411834]: Next returned nil segment Dec 25 04:40:41.172473798 (469h11m41.043556833s) [upload=411834]: Waiting for error group managing segments... Dec 25 04:40:41.483361556 (469h11m41.354444601s) [noise=true , address=145.79.227.37:4567 , piece_id=TZ4C57ISUI3CB3VOLTSM5W5QVDI7PVXV67PFJI3ZSVSXHTKPYPSA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.484283170 (469h11m41.355366205s) [noise=true , address=145.79.227.5:4221 , piece_id=C4SXUVTG2TZ4EIRBGL4TLOLBMYOAZ4KLIQSZOXKEOWAJBQQJLWSA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.515393699 (469h11m41.386476744s) [noise=true , address=145.79.227.51:4411 , piece_id=WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.526179387 (469h11m41.397262422s) [noise=true , address=45.91.116.5:4300 , piece_id=NTWC3RWYQQBBKPMXGUG44R2B5GWC2Z4LDFU3GTWOWCT3JIRIZOOQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.642380892 (469h11m41.513463937s) [noise=true , address=145.79.227.36:4517 , piece_id=HBQU7CMAO2KJS6ODW2WHICBAWQSH6TQQPOQSENKDEISWCWFN2CMA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.662326662 (469h11m41.533409707s) [noise=true , address=145.79.227.3:4150 , piece_id=OHGA4EA33AVEDCJLB6QSUJLD3W2TV6YNN6FBVFFKTTSF2B6XDPRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.758852130 (469h11m41.629935175s) [noise=true , address=145.79.227.47:4265 , piece_id=TVUXNR6K35A3CWFDN4JRKM4Q6RGBC4LP2XKPXJFN2AJL32ESEBPA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.823449082 (469h11m41.694532117s) [noise=true , address=145.79.227.17:4970 , piece_id=6DPKJ762IUN4A7UEZXLDERCVDW24QEKGEHFETAYZOKWQPVN2QJYA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.952636616 (469h11m41.823719661s) [noise=true , address=145.79.227.19:4079 , piece_id=ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.025177123 (469h11m41.896260168s) [noise=true , address=145.79.227.45:4229 , piece_id=GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.222526274 (469h11m42.093609319s) [noise=true , address=145.79.227.8:4331 , piece_id=ELLVNJFOVY5HSAHZDQYGQVO7OYHFZNFHCBJKHQFV5K6EV2M3JTRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.250024374 (469h11m42.121107409s) [noise=true , address=145.79.227.2:4114 , piece_id=5ISXZKQSH4CLLTF3ILGAXHV6STMLRUSZP6UATU2677X2FUPLBCRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.651787921 (469h11m42.522870966s) [noise=true , address=145.79.227.15:4875 , piece_id=FUMLT2ODHUE3ETSNWVKBYLSJXYVCHUW5KAJJE5AECNL3IAQYYCLQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.651814982 (469h11m42.522898027s) [seg_pos={0 0}, upload=411834]: Segment reached optimal threshold of 13 pieces. Dec 25 04:40:42.652052093 (469h11m42.523135129s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: write tcp 188.166.225.170:36650->77.90.31.15:4697: use of closed network connection; storage node overloaded, request limit: 2000 Dec 25 04:40:42.652069537 (469h11m42.523152572s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652177673 (469h11m42.523260708s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd): protocol: write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection; write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection Dec 25 04:40:42.652184456 (469h11m42.523267501s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652230243 (469h11m42.523313308s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq): protocol: write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection Dec 25 04:40:42.652240693 (469h11m42.523323778s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652268265 (469h11m42.52335131s) [seg_pos={0 0}, upload=411834]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 04:40:42.652354770 (469h11m42.523437815s) [upload=411834]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 04:40:42.652361693 (469h11m42.523444738s) [upload=411834]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 04:40:42.959704440 (469h11m42.830787485s) [upload=411834]: Done uploading segments... Dec 25 04:40:42.959728234 (469h11m42.83081127s) [upload=411834]: Upload finished. err: <nil> Duration: 6.12216021s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 5.921992453s trace: https://jaeger.dev.storj.io/trace/681cf842d2260d41 FAIL (2025-12-25 05:00:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, Piece ID: ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ) context deadline exceeded; piecestore: (Node ID: 1DT68m6J4PpDk3XUDhDaaqV46NLKVz4A23716JLFxEAh122GeR, Piece ID: WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA) context deadline exceeded; piecestore: (Node ID: 12wWYnhWHSqqAFgje4p9Cr7WYaJ6dJWsCqvrx8ZqVFoakjuFeka, Piece ID: GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q) context deadline exceeded ======LOGS====== nline: false Dec 25 04:40:39.039085560 (469h11m38.910168605s) [upload=411834]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 04:40:39.039095028 (469h11m38.910178073s) [upload=411834]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 04:40:39.297407613 (469h11m39.168490658s) [seg_pos={0 0}, upload=411834]: Begin upload segment... Dec 25 04:40:39.297471084 (469h11m39.168554119s) [seg_pos={0 0}, upload=411834]: Done begin upload segment. Dec 25 04:40:39.297511581 (469h11m39.168594616s) [noise=true , address=145.79.227.51:4411 , piece_id=WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.297663310 (469h11m39.168746345s) [noise=true , address=145.79.227.17:4970 , piece_id=6DPKJ762IUN4A7UEZXLDERCVDW24QEKGEHFETAYZOKWQPVN2QJYA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.298405574 (469h11m39.169488619s) [noise=true , address=145.79.227.2:4114 , piece_id=5ISXZKQSH4CLLTF3ILGAXHV6STMLRUSZP6UATU2677X2FUPLBCRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.298797961 (469h11m39.169881006s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299124392 (469h11m39.170207437s) [noise=true , address=45.91.116.5:4300 , piece_id=NTWC3RWYQQBBKPMXGUG44R2B5GWC2Z4LDFU3GTWOWCT3JIRIZOOQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299438650 (469h11m39.170521685s) [noise=true , address=145.79.227.5:4221 , piece_id=C4SXUVTG2TZ4EIRBGL4TLOLBMYOAZ4KLIQSZOXKEOWAJBQQJLWSA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.299794709 (469h11m39.170877744s) [noise=true , address=145.79.227.37:4567 , piece_id=TZ4C57ISUI3CB3VOLTSM5W5QVDI7PVXV67PFJI3ZSVSXHTKPYPSA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.300142221 (469h11m39.171225256s) [noise=true , address=145.79.227.8:4331 , piece_id=ELLVNJFOVY5HSAHZDQYGQVO7OYHFZNFHCBJKHQFV5K6EV2M3JTRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.300506354 (469h11m39.171589389s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.301142795 (469h11m39.17222584s) [noise=true , address=145.79.227.36:4517 , piece_id=HBQU7CMAO2KJS6ODW2WHICBAWQSH6TQQPOQSENKDEISWCWFN2CMA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.301625835 (469h11m39.17270891s) [noise=true , address=145.79.227.45:4229 , piece_id=GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.302170964 (469h11m39.173254009s) [noise=true , address=145.79.227.3:4150 , piece_id=OHGA4EA33AVEDCJLB6QSUJLD3W2TV6YNN6FBVFFKTTSF2B6XDPRA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.302629947 (469h11m39.173712982s) [noise=true , address=145.79.227.19:4079 , piece_id=ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303079704 (469h11m39.174162749s) [noise=true , address=145.79.227.15:4875 , piece_id=FUMLT2ODHUE3ETSNWVKBYLSJXYVCHUW5KAJJE5AECNL3IAQYYCLQ, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303499804 (469h11m39.174582849s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:39.303921386 (469h11m39.175004432s) [noise=true , address=145.79.227.47:4265 , piece_id=TVUXNR6K35A3CWFDN4JRKM4Q6RGBC4LP2XKPXJFN2AJL32ESEBPA, seg_pos={0 0}, upload=411834]: Uploading piece... Dec 25 04:40:41.172453460 (469h11m41.043536495s) [upload=411834]: Next returned nil segment Dec 25 04:40:41.172473798 (469h11m41.043556833s) [upload=411834]: Waiting for error group managing segments... Dec 25 04:40:41.483361556 (469h11m41.354444601s) [noise=true , address=145.79.227.37:4567 , piece_id=TZ4C57ISUI3CB3VOLTSM5W5QVDI7PVXV67PFJI3ZSVSXHTKPYPSA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.484283170 (469h11m41.355366205s) [noise=true , address=145.79.227.5:4221 , piece_id=C4SXUVTG2TZ4EIRBGL4TLOLBMYOAZ4KLIQSZOXKEOWAJBQQJLWSA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.515393699 (469h11m41.386476744s) [noise=true , address=145.79.227.51:4411 , piece_id=WFELZUWR3OHDE4J4UA3NY7CHQN245NIDLATYFBII5UYNAONT4YKA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.526179387 (469h11m41.397262422s) [noise=true , address=45.91.116.5:4300 , piece_id=NTWC3RWYQQBBKPMXGUG44R2B5GWC2Z4LDFU3GTWOWCT3JIRIZOOQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.642380892 (469h11m41.513463937s) [noise=true , address=145.79.227.36:4517 , piece_id=HBQU7CMAO2KJS6ODW2WHICBAWQSH6TQQPOQSENKDEISWCWFN2CMA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.662326662 (469h11m41.533409707s) [noise=true , address=145.79.227.3:4150 , piece_id=OHGA4EA33AVEDCJLB6QSUJLD3W2TV6YNN6FBVFFKTTSF2B6XDPRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.758852130 (469h11m41.629935175s) [noise=true , address=145.79.227.47:4265 , piece_id=TVUXNR6K35A3CWFDN4JRKM4Q6RGBC4LP2XKPXJFN2AJL32ESEBPA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.823449082 (469h11m41.694532117s) [noise=true , address=145.79.227.17:4970 , piece_id=6DPKJ762IUN4A7UEZXLDERCVDW24QEKGEHFETAYZOKWQPVN2QJYA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:41.952636616 (469h11m41.823719661s) [noise=true , address=145.79.227.19:4079 , piece_id=ODEKOAZU2RZWWXRMHQMGRX5YREEYMMHIYKBAPTHHY65Q47FV7ZIQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.025177123 (469h11m41.896260168s) [noise=true , address=145.79.227.45:4229 , piece_id=GGKWZ24EPBZC5K3JOZOHTR4WHN663AJ2YQAPHKCCSKUEDTC2RE6Q, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.222526274 (469h11m42.093609319s) [noise=true , address=145.79.227.8:4331 , piece_id=ELLVNJFOVY5HSAHZDQYGQVO7OYHFZNFHCBJKHQFV5K6EV2M3JTRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.250024374 (469h11m42.121107409s) [noise=true , address=145.79.227.2:4114 , piece_id=5ISXZKQSH4CLLTF3ILGAXHV6STMLRUSZP6UATU2677X2FUPLBCRA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.651787921 (469h11m42.522870966s) [noise=true , address=145.79.227.15:4875 , piece_id=FUMLT2ODHUE3ETSNWVKBYLSJXYVCHUW5KAJJE5AECNL3IAQYYCLQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: <nil> Dec 25 04:40:42.651814982 (469h11m42.522898027s) [seg_pos={0 0}, upload=411834]: Segment reached optimal threshold of 13 pieces. Dec 25 04:40:42.652052093 (469h11m42.523135129s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: write tcp 188.166.225.170:36650->77.90.31.15:4697: use of closed network connection; storage node overloaded, request limit: 2000 Dec 25 04:40:42.652069537 (469h11m42.523152572s) [noise=true , address=77.90.31.15:4697 , piece_id=HT7XSGPLBDGCU77SKQJUTPGRAANZ2EW22BSN4XENKBDMMIZUDAZQ, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652177673 (469h11m42.523260708s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd): protocol: write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection; write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39288->145.79.227.39:4014: use of closed network connection Dec 25 04:40:42.652184456 (469h11m42.523267501s) [noise=true , address=145.79.227.39:4014 , piece_id=PDIX3KCPKZPH5VLWWARRVZMN5I2PONL5MDGHW3QAB432EB57MJ4Q, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652230243 (469h11m42.523313308s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq): protocol: write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47434->145.79.227.9:4365: use of closed network connection Dec 25 04:40:42.652240693 (469h11m42.523323778s) [noise=true , address=145.79.227.9:4365 , piece_id=PYGRTZ3R2Y7H6HPHDLLQULNI2Y7Q2CA5GWZE4V3FYWHUQ5X3AGDA, seg_pos={0 0}, upload=411834]: Long tail cancellation due to optimal threshold Dec 25 04:40:42.652268265 (469h11m42.52335131s) [seg_pos={0 0}, upload=411834]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 04:40:42.652354770 (469h11m42.523437815s) [upload=411834]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 04:40:42.652361693 (469h11m42.523444738s) [upload=411834]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 04:40:42.959704440 (469h11m42.830787485s) [upload=411834]: Done uploading segments... Dec 25 04:40:42.959728234 (469h11m42.83081127s) [upload=411834]: Upload finished. err: <nil> Duration: 5.922025615s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)