Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 6.416380015s trace: https://jaeger.dev.storj.io/trace/572c91a59c970da7 FAIL (2025-12-05 14:31:50 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 6.416407466s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 45 out of 60 is failed (threshold=6) [XXXXXXXXXXXX.X..XXXXX..XXXXXX.X.XXXXXXX.XXX.XXXXX.XXX.X....X] duration: 5.001778053s trace: https://jaeger.dev.storj.io/trace/55d21f6a82e1e4af FAIL (2025-12-05 14:31:39 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00182258s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 7.559269489s trace: https://jaeger.dev.storj.io/trace/572c91a59c970da7 FAIL (2025-12-05 14:31:26 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 7.559303463s core uplink us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....X....................................X..................] duration: 15.124865645s trace: https://jaeger.dev.storj.io/trace/496d6a85c2db53eb FAIL (2025-12-05 14:31:23 UTC) orbiter warning: context deadline exceeded Duration: 15.124946851s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 2.872953862s trace: https://jaeger.dev.storj.io/trace/41b9575307efc1e3 FAIL (2025-12-05 14:31:16 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 2.872983589s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 2.816853657s trace: https://jaeger.dev.storj.io/trace/361a2ea17a62744e FAIL (2025-12-05 14:31:15 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 5 14:08:02.649750072 (45h12m39.120960811s) [upload=39762]: Starting upload Dec 5 14:08:02.650012922 (45h12m39.121223681s) [upload=39762]: Done starting upload Dec 5 14:08:02.650140385 (45h12m39.121351124s) [upload=39762]: Uploading segments... Dec 5 14:08:02.650204036 (45h12m39.121414775s) [upload=39762]: Got next segment. Inline: true Dec 5 14:08:02.650210659 (45h12m39.121421398s) [upload=39762]: Waiting for error group managing segments... Dec 5 14:08:02.650285611 (45h12m39.12149635s) [upload=39762]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 5 14:08:02.650292053 (45h12m39.121502792s) [upload=39762]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Dec 5 14:08:02.650295811 (45h12m39.12150655s) [upload=39762]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 5 14:08:05.223576463 (45h12m41.694787212s) [upload=39762]: Done uploading segments... Dec 5 14:08:05.223604205 (45h12m41.694814934s) [upload=39762]: Upload finished. err: <nil> Duration:* 2.81688706s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 9.525361792s trace: https://jaeger.dev.storj.io/trace/65df260343fa2062 FAIL (2025-12-05 14:31:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, Piece ID: 2KY6XA6532G23SRYSNBYNYZ2Y65ZLMXDC2ERGQ6PH4UJQZDUZRUA) context deadline exceeded; piecestore: (Node ID: 12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, Piece ID: JYAT6OFRJQNEZPORFRHUCTFQDPVM2ZTXEUBWGNY6BLADCTZIS66Q) context deadline exceeded; piecestore: (Node ID: 1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, Piece ID: RV4CB7VT7T3YVFFBGXA6EVYMGEOWBU3C3XYUTEMRZDBVHRNVEVLA) context deadline exceeded; piecestore: (Node ID: 12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, Piece ID: VMP5DCNJDH5EAO6LOHSOCP4GGOYI6EV3YHCW5GXB2BRTOO67YQMQ) context deadline exceeded; piecestore: (Node ID: 1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5, Piece ID: UNWONHZ6EIRFROZ5L5YAEFBSMFZHEN4VKAKYM2DWTW4BGCTNMAMA) context deadline exceeded ======LOGS====== Dec 5 13:50:13.914958542 (44h54m50.386169281s) [upload=39497]: Starting upload Dec 5 13:50:13.915042042 (44h54m50.386252771s) [upload=39497]: Done starting upload Dec 5 13:50:13.915237493 (44h54m50.386448232s) [upload=39497]: Uploading segments... Dec 5 13:50:13.915339918 (44h54m50.386550688s) [upload=39497]: Got next segment. Inline: false Dec 5 13:50:13.915395575 (44h54m50.386606314s) [upload=39497]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 5 13:50:13.915400735 (44h54m50.386611464s) [upload=39497]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 5 13:50:14.177967782 (44h54m50.649178531s) [seg_pos={0 0}, upload=39497]: Begin upload segment... Dec 5 13:50:14.178049498 (44h54m50.649260237s) [seg_pos={0 0}, upload=39497]: Done begin upload segment. Dec 5 13:50:14.178089053 (44h54m50.649299792s) [noise=true , address=145.79.227.34:4906 , piece_id=L2SWU22TLNZNS3LO2E3HYCKOVPO6CIOZWOACPQV7CGMURN2HEQRQ, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.178100966 (44h54m50.649311715s) [noise=true , address=145.79.227.10:4402 , piece_id=IHNZYV6LJREQ63GBFJY6BVB54QLSVCZIIUUTLOG7YOOOXNGHQ7WA, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.178155129 (44h54m50.649365868s) [noise=true , address=145.79.227.23:4546 , piece_id=UNWONHZ6EIRFROZ5L5YAEFBSMFZHEN4VKAKYM2DWTW4BGCTNMAMA, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.178737678 (44h54m50.649948417s) [noise=true , address=145.79.227.39:4015 , piece_id=V6WH4XCY2YVP2B3DR4HLI7IFH7MGGPAIBAZHHDBHQQ4NUTCNHP7Q, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.179277977 (44h54m50.650488716s) [noise=true , address=145.79.227.21:4474 , piece_id=J5V4QU72P7D2FGZ7NWO5UT3E7S25CTN4MCSPQMSOWK7R6GK4B4HA, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.179826452 (44h54m50.651037191s) [noise=true , address=145.79.227.21:4475 , piece_id=5IPJJ5RHSXCJPUUXXD62EKMNSWSHPIVEWWYS4X77GE7OT6NCTJUA, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.180516445 (44h54m50.651727174s) [noise=true , address=145.79.227.38:4978 , piece_id=N7OVLKMMMZW75CCZDX5QYOG2SDCO7AKO2FITHFDO3EMDSPJJPAYA, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.181184327 (44h54m50.652395066s) [noise=true , address=145.79.227.43:4157 , piece_id=PFPPMUR22PUIHIPCNXQGN5CYFM4K4OG2EMC5LAMUH26RZKETHZAQ, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.182617817 (44h54m50.653828556s) [noise=true , address=145.79.227.41:4087 , piece_id=2KY6XA6532G23SRYSNBYNYZ2Y65ZLMXDC2ERGQ6PH4UJQZDUZRUA, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.183276120 (44h54m50.654486859s) [noise=true , address=145.79.227.16:4909 , piece_id=JYAT6OFRJQNEZPORFRHUCTFQDPVM2ZTXEUBWGNY6BLADCTZIS66Q, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.184534556 (44h54m50.655745295s) [noise=true , address=145.79.227.20:4438 , piece_id=P64J3PKA4V3SQWAC36KWBVXDBE4KXYJHUXWYHNFZHCQEGQZUWHFQ, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.186480883 (44h54m50.657691622s) [noise=true , address=145.79.227.25:4619 , piece_id=RV4CB7VT7T3YVFFBGXA6EVYMGEOWBU3C3XYUTEMRZDBVHRNVEVLA, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.188448450 (44h54m50.659659189s) [noise=true , address=77.90.31.14:4663 , piece_id=7WW2DVMZSNJTG3J4NANUAIZKMLICAYLP5RAOKKWCTCUTJAISG7SQ, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.190696029 (44h54m50.661906768s) [noise=true , address=145.79.227.4:4187 , piece_id=VMP5DCNJDH5EAO6LOHSOCP4GGOYI6EV3YHCW5GXB2BRTOO67YQMQ, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.193175661 (44h54m50.6643864s) [noise=true , address=145.79.227.31:4799 , piece_id=2C63VGDP7JVQVWSLYLHZEHC4SB7PU6NFUZWLCTEO7AGDP32X3JOQ, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.195649231 (44h54m50.66685997s) [noise=true , address=145.79.227.14:4838 , piece_id=6EVD7RSXQJOFT645D7ILWLUWUBYFP6IRR5GESYOAWXF2IXKJEUCA, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:15.979218426 (44h54m52.450429165s) [upload=39497]: Next returned nil segment Dec 5 13:50:15.979231121 (44h54m52.45044185s) [upload=39497]: Waiting for error group managing segments... Dec 5 13:50:16.323526524 (44h54m52.794737263s) [noise=true , address=145.79.227.21:4475 , piece_id=5IPJJ5RHSXCJPUUXXD62EKMNSWSHPIVEWWYS4X77GE7OT6NCTJUA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:16.347685978 (44h54m52.818896717s) [noise=true , address=145.79.227.23:4546 , piece_id=UNWONHZ6EIRFROZ5L5YAEFBSMFZHEN4VKAKYM2DWTW4BGCTNMAMA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:16.351351307 (44h54m52.822562036s) [noise=true , address=145.79.227.10:4402 , piece_id=IHNZYV6LJREQ63GBFJY6BVB54QLSVCZIIUUTLOG7YOOOXNGHQ7WA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:16.351375874 (44h54m52.822586603s) [noise=true , address=145.79.227.38:4978 , piece_id=N7OVLKMMMZW75CCZDX5QYOG2SDCO7AKO2FITHFDO3EMDSPJJPAYA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:16.359501778 (44h54m52.830712517s) [noise=true , address=145.79.227.31:4799 , piece_id=2C63VGDP7JVQVWSLYLHZEHC4SB7PU6NFUZWLCTEO7AGDP32X3JOQ, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:16.360729425 (44h54m52.831940164s) [noise=true , address=145.79.227.4:4187 , piece_id=VMP5DCNJDH5EAO6LOHSOCP4GGOYI6EV3YHCW5GXB2BRTOO67YQMQ, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:16.371054476 (44h54m52.842265225s) [noise=true , address=145.79.227.41:4087 , piece_id=2KY6XA6532G23SRYSNBYNYZ2Y65ZLMXDC2ERGQ6PH4UJQZDUZRUA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:16.392350999 (44h54m52.863561738s) [noise=true , address=77.90.31.14:4663 , piece_id=7WW2DVMZSNJTG3J4NANUAIZKMLICAYLP5RAOKKWCTCUTJAISG7SQ, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:16.437887268 (44h54m52.909098007s) [noise=true , address=145.79.227.21:4474 , piece_id=J5V4QU72P7D2FGZ7NWO5UT3E7S25CTN4MCSPQMSOWK7R6GK4B4HA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:16.642787347 (44h54m53.113998086s) [noise=true , address=145.79.227.43:4157 , piece_id=PFPPMUR22PUIHIPCNXQGN5CYFM4K4OG2EMC5LAMUH26RZKETHZAQ, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:16.987804677 (44h54m53.459015416s) [noise=true , address=145.79.227.25:4619 , piece_id=RV4CB7VT7T3YVFFBGXA6EVYMGEOWBU3C3XYUTEMRZDBVHRNVEVLA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:17.014774723 (44h54m53.485985462s) [noise=true , address=145.79.227.14:4838 , piece_id=6EVD7RSXQJOFT645D7ILWLUWUBYFP6IRR5GESYOAWXF2IXKJEUCA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:17.413923466 (44h54m53.885134205s) [noise=true , address=145.79.227.16:4909 , piece_id=JYAT6OFRJQNEZPORFRHUCTFQDPVM2ZTXEUBWGNY6BLADCTZIS66Q, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:17.413981627 (44h54m53.885192367s) [seg_pos={0 0}, upload=39497]: Segment reached optimal threshold of 13 pieces. Dec 5 13:50:17.414271349 (44h54m53.885482088s) [noise=true , address=145.79.227.39:4015 , piece_id=V6WH4XCY2YVP2B3DR4HLI7IFH7MGGPAIBAZHHDBHQQ4NUTCNHP7Q, seg_pos={0 0}, upload=39497]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE): protocol: write tcp 188.166.225.170:56230->145.79.227.39:4015: use of closed network connection; write tcp 188.166.225.170:56230->145.79.227.39:4015: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56230->145.79.227.39:4015: use of closed network connection Dec 5 13:50:17.414282390 (44h54m53.885493119s) [noise=true , address=145.79.227.39:4015 , piece_id=V6WH4XCY2YVP2B3DR4HLI7IFH7MGGPAIBAZHHDBHQQ4NUTCNHP7Q, seg_pos={0 0}, upload=39497]: Long tail cancellation due to optimal threshold Dec 5 13:50:17.414355008 (44h54m53.885565747s) [noise=true , address=145.79.227.34:4906 , piece_id=L2SWU22TLNZNS3LO2E3HYCKOVPO6CIOZWOACPQV7CGMURN2HEQRQ, seg_pos={0 0}, upload=39497]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 5 13:50:17.414365318 (44h54m53.885576057s) [noise=true , address=145.79.227.34:4906 , piece_id=L2SWU22TLNZNS3LO2E3HYCKOVPO6CIOZWOACPQV7CGMURN2HEQRQ, seg_pos={0 0}, upload=39497]: Long tail cancellation due to optimal threshold Dec 5 13:50:17.414811999 (44h54m53.886022758s) [noise=true , address=145.79.227.20:4438 , piece_id=P64J3PKA4V3SQWAC36KWBVXDBE4KXYJHUXWYHNFZHCQEGQZUWHFQ, seg_pos={0 0}, upload=39497]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT): protocol: expected piece hash; context canceled; EOF Dec 5 13:50:17.414824472 (44h54m53.886035211s) [noise=true , address=145.79.227.20:4438 , piece_id=P64J3PKA4V3SQWAC36KWBVXDBE4KXYJHUXWYHNFZHCQEGQZUWHFQ, seg_pos={0 0}, upload=39497]: Long tail cancellation due to optimal threshold Dec 5 13:50:17.414850933 (44h54m53.886061672s) [seg_pos={0 0}, upload=39497]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 5 13:50:17.414965771 (44h54m53.88617652s) [upload=39497]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 5 13:50:17.414973846 (44h54m53.886184585s) [upload=39497]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 5 13:50:17.720870703 (44h54m54.192081452s) [upload=39497]: Done uploading segments... Dec 5 13:50:17.720894969 (44h54m54.192105708s) [upload=39497]: Upload finished. err: <nil> Duration: 9.525401237s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 10.612647746s trace: https://jaeger.dev.storj.io/trace/70e54e58d52fa95e FAIL (2025-12-05 14:31:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1MntjJtcCDSRozKpesp6S2ZjYrMhxiQheDiPMPSCnavWBBGYt3, Piece ID: MQGKTFSYGQSOMNUEIVOGGS5ZEVM52CLW4TEG7DA5ICOK73GEAI5Q) context deadline exceeded; piecestore: (Node ID: 12YnWetw1NereHQi7cwHCnmu1LDjRFNpB2MQHSWQrjQdN1CZbMw, Piece ID: UALY4LTNPIS62L46OIH4PTKRES72I7LA55UL7B2UCZLRT7DQ7RVQ) context deadline exceeded; piecestore: (Node ID: 12Dfm6iHQbMhj1YWLu3UboTCtZ2hq87aiYUgH7Kig3pDUrPeFj4, Piece ID: 4W72IA43JPPLQ5ZDZHZDW6AFEZDVG7AHJT7555P4BY2Z3QYMLKNA) context deadline exceeded; piecestore: (Node ID: 12Qb6sMhCyzYd4S15Td1kzD7kgqNKSi2avG15CQRSG8bcDVvGzi, Piece ID: SUX7UBSIQIIYU327MGSXKLJBPXFW6UEVI3NJRAWVEZ3GBUN2QCJQ) context deadline exceeded; piecestore: (Node ID: 1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks, Piece ID: WBCLEOB7XBAT2MPT6M7NLLNQM2JR3S22BV6UZBSGRDFIJP5JS2LA) context deadline exceeded; piecestore: (Node ID: 123QEF1LQm8nDCTjZvq51BjSMTYwHjhj2ajZd7bCRjiWJVLGXyS, Piece ID: MBHG2UWXQMEZPJJHW67KJMJTIVT4QWPXKDIRBRVTBAFN46J3M42Q) context deadline exceeded; piecestore: (Node ID: 1gPa2A8V3hY6WFgDTMk5ZMQpEaVVuhjXFQYHoZhK2R9JUPyepz, Piece ID: ZS3JXULFI4AGS3HIIX2POYQ3XLBG6GPFAWBAV2NBRLMO447DH7ZA) context deadline exceeded; piecestore: (Node ID: 121RT7tP6czMFAiZ3DjzSSsYgPt3xZXVbXVpkcYxaZYscNiMjL6, Piece ID: EPHUILAEI77HCSYICLO2IMEGVISUXUEQGKNAHV52VDAGDCAP3RKQ) context deadline exceeded; piecestore: (Node ID: 1if7vf1cNTBwAhb6owi6tuGLYqSfa3feWhsvGSevwuwykhz2RY, Piece ID: CFMB4CDU7DWOF3L5T7DASMJB33RVAYSTMM4WFNUSUNV4H6CGPNXQ) context deadline exceeded; piecestore: (Node ID: 1ArQdwHzHSm7c2QZkTt1kQsfzocvCNG2D9yoQgnmbS7HYLCXhy, Piece ID: IY7G4GQVUEYHNO4HJAD3CHGX4SQQNGWPMUXUJJKLC7MUTUUFGFWA) context deadline exceeded ======LOGS====== 4909 , piece_id=EPHUILAEI77HCSYICLO2IMEGVISUXUEQGKNAHV52VDAGDCAP3RKQ, seg_pos={0 0}, upload=39772]: Uploading piece... Dec 5 14:09:11.669180895 (45h13m48.140391634s) [noise=true , address=77.90.31.12:4616 , piece_id=SUX7UBSIQIIYU327MGSXKLJBPXFW6UEVI3NJRAWVEZ3GBUN2QCJQ, seg_pos={0 0}, upload=39772]: Uploading piece... Dec 5 14:09:11.669641381 (45h13m48.14085212s) [noise=true , address=145.79.227.32:4855 , piece_id=N3F2LVAV3UNBWZ7V7HFLNVU5NGWD6X2RB6WKVEVGBJMHUUIJP3PA, seg_pos={0 0}, upload=39772]: Uploading piece... Dec 5 14:09:11.670111447 (45h13m48.141322196s) [noise=true , address=77.90.31.17:4794 , piece_id=ALZXJQTN64GGUOZPJMGJ6CRXQBO2R5PRVNUBQHRKWYGY7KJF6CVQ, seg_pos={0 0}, upload=39772]: Uploading piece... Dec 5 14:09:11.670601259 (45h13m48.141811999s) [noise=true , address=64.49.13.169:4705 , piece_id=TTCHF4ENHBWLOJQCAJKMG6BUZGHDDEU3KW5AU5Q3CFCOTKIONE4Q, seg_pos={0 0}, upload=39772]: Uploading piece... Dec 5 14:09:11.671075502 (45h13m48.142286241s) [noise=true , address=77.90.31.13:4654 , piece_id=CEAQAXPEBEPYGOMBFNVRRYRCT67SMMNEVDBADWIOPQXEJCN3EGZQ, seg_pos={0 0}, upload=39772]: Uploading piece... Dec 5 14:09:11.671598037 (45h13m48.142808776s) [noise=true , address=45.91.116.11:4147 , piece_id=UALY4LTNPIS62L46OIH4PTKRES72I7LA55UL7B2UCZLRT7DQ7RVQ, seg_pos={0 0}, upload=39772]: Uploading piece... Dec 5 14:09:11.672127346 (45h13m48.143338105s) [noise=true , address=145.79.227.15:4907 , piece_id=BXEPFHK6AENJZM2APP5LPV2LHJ4L7CBFQF5JSQJQBC6F5AKDGYQA, seg_pos={0 0}, upload=39772]: Uploading piece... Dec 5 14:09:11.672648588 (45h13m48.143859327s) [noise=true , address=77.90.31.18:4838 , piece_id=CFMB4CDU7DWOF3L5T7DASMJB33RVAYSTMM4WFNUSUNV4H6CGPNXQ, seg_pos={0 0}, upload=39772]: Uploading piece... Dec 5 14:09:11.673155192 (45h13m48.144365931s) [noise=true , address=77.90.31.19:4856 , piece_id=MQGKTFSYGQSOMNUEIVOGGS5ZEVM52CLW4TEG7DA5ICOK73GEAI5Q, seg_pos={0 0}, upload=39772]: Uploading piece... Dec 5 14:09:11.673691684 (45h13m48.144902423s) [noise=true , address=45.91.116.10:4091 , piece_id=ZS3JXULFI4AGS3HIIX2POYQ3XLBG6GPFAWBAV2NBRLMO447DH7ZA, seg_pos={0 0}, upload=39772]: Uploading piece... Dec 5 14:09:11.674805966 (45h13m48.146016705s) [noise=true , address=145.79.227.19:4097 , piece_id=MKAKS25RPIEVHT2WJJ7QYY4FBS7JCIM3RKY7MWRJ7FZTZ2YZXMGQ, seg_pos={0 0}, upload=39772]: Uploading piece... Dec 5 14:09:11.675360772 (45h13m48.146571511s) [noise=true , address=45.91.116.5:4329 , piece_id=MBHG2UWXQMEZPJJHW67KJMJTIVT4QWPXKDIRBRVTBAFN46J3M42Q, seg_pos={0 0}, upload=39772]: Uploading piece... Dec 5 14:09:13.239845356 (45h13m49.711056085s) [upload=39772]: Next returned nil segment Dec 5 14:09:13.239860604 (45h13m49.711071334s) [upload=39772]: Waiting for error group managing segments... Dec 5 14:09:13.643777330 (45h13m50.114988069s) [noise=true , address=77.90.31.13:4654 , piece_id=CEAQAXPEBEPYGOMBFNVRRYRCT67SMMNEVDBADWIOPQXEJCN3EGZQ, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil> Dec 5 14:09:13.697999800 (45h13m50.169210539s) [noise=true , address=77.90.31.18:4838 , piece_id=CFMB4CDU7DWOF3L5T7DASMJB33RVAYSTMM4WFNUSUNV4H6CGPNXQ, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil> Dec 5 14:09:13.713127946 (45h13m50.184338685s) [noise=true , address=77.90.31.15:4729 , piece_id=IY7G4GQVUEYHNO4HJAD3CHGX4SQQNGWPMUXUJJKLC7MUTUUFGFWA, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil> Dec 5 14:09:13.718498752 (45h13m50.189709491s) [noise=true , address=77.90.31.21:4909 , piece_id=EPHUILAEI77HCSYICLO2IMEGVISUXUEQGKNAHV52VDAGDCAP3RKQ, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil> Dec 5 14:09:13.726522451 (45h13m50.19773318s) [noise=true , address=77.90.31.17:4794 , piece_id=ALZXJQTN64GGUOZPJMGJ6CRXQBO2R5PRVNUBQHRKWYGY7KJF6CVQ, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil> Dec 5 14:09:13.749057442 (45h13m50.220268171s) [noise=true , address=45.91.116.10:4091 , piece_id=ZS3JXULFI4AGS3HIIX2POYQ3XLBG6GPFAWBAV2NBRLMO447DH7ZA, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil> Dec 5 14:09:13.750808776 (45h13m50.222019515s) [noise=true , address=77.90.31.10:4542 , piece_id=XDD4QZAKCWJK5A3F5FDL4NRNPEQZF3GKODCGVT2V47AH6KVEOOWQ, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil> Dec 5 14:09:13.754762756 (45h13m50.225973485s) [noise=true , address=77.90.31.11:4556 , piece_id=WBCLEOB7XBAT2MPT6M7NLLNQM2JR3S22BV6UZBSGRDFIJP5JS2LA, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil> Dec 5 14:09:13.759242185 (45h13m50.230452925s) [noise=true , address=77.90.31.12:4616 , piece_id=SUX7UBSIQIIYU327MGSXKLJBPXFW6UEVI3NJRAWVEZ3GBUN2QCJQ, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil> Dec 5 14:09:13.774028982 (45h13m50.245239711s) [noise=true , address=77.90.31.19:4856 , piece_id=MQGKTFSYGQSOMNUEIVOGGS5ZEVM52CLW4TEG7DA5ICOK73GEAI5Q, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil> Dec 5 14:09:13.786684680 (45h13m50.257895419s) [noise=true , address=45.91.116.9:4272 , piece_id=4W72IA43JPPLQ5ZDZHZDW6AFEZDVG7AHJT7555P4BY2Z3QYMLKNA, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil> Dec 5 14:09:13.790669208 (45h13m50.261879947s) [noise=true , address=45.91.116.11:4147 , piece_id=UALY4LTNPIS62L46OIH4PTKRES72I7LA55UL7B2UCZLRT7DQ7RVQ, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil> Dec 5 14:09:13.850747877 (45h13m50.321958617s) [noise=true , address=45.91.116.8:4071 , piece_id=3YI67CEXFW75VRROYLSABR55DNVFHE4SR6EUP57CPRIIB63GZOVA, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil> Dec 5 14:09:13.884817071 (45h13m50.3560278s) [noise=true , address=45.91.116.5:4329 , piece_id=MBHG2UWXQMEZPJJHW67KJMJTIVT4QWPXKDIRBRVTBAFN46J3M42Q, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil> Dec 5 14:09:13.884838361 (45h13m50.35604909s) [seg_pos={0 0}, upload=39772]: Segment reached optimal threshold of 14 pieces. Dec 5 14:09:13.885054584 (45h13m50.356265313s) [noise=true , address=145.79.227.15:4907 , piece_id=BXEPFHK6AENJZM2APP5LPV2LHJ4L7CBFQF5JSQJQBC6F5AKDGYQA, seg_pos={0 0}, upload=39772]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JNee734A635XWMM3Q3N6siDsvKY3FEMQerDjYZrQR6orGBaY3): protocol: write tcp 188.166.225.170:59612->145.79.227.15:4907: use of closed network connection; write tcp 188.166.225.170:59612->145.79.227.15:4907: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59612->145.79.227.15:4907: use of closed network connection Dec 5 14:09:13.885064302 (45h13m50.356275041s) [noise=true , address=145.79.227.15:4907 , piece_id=BXEPFHK6AENJZM2APP5LPV2LHJ4L7CBFQF5JSQJQBC6F5AKDGYQA, seg_pos={0 0}, upload=39772]: Long tail cancellation due to optimal threshold Dec 5 14:09:13.885416693 (45h13m50.356627432s) [noise=true , address=145.79.227.19:4097 , piece_id=MKAKS25RPIEVHT2WJJ7QYY4FBS7JCIM3RKY7MWRJ7FZTZ2YZXMGQ, seg_pos={0 0}, upload=39772]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QP2aZ9BC22cD1Nzz2E12DiA1XoNu3GGTgHC8Cc3TvpQM3Ubn3): protocol: write tcp 188.166.225.170:37344->145.79.227.19:4097: use of closed network connection; write tcp 188.166.225.170:37344->145.79.227.19:4097: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37344->145.79.227.19:4097: use of closed network connection Dec 5 14:09:13.885427744 (45h13m50.356638483s) [noise=true , address=145.79.227.19:4097 , piece_id=MKAKS25RPIEVHT2WJJ7QYY4FBS7JCIM3RKY7MWRJ7FZTZ2YZXMGQ, seg_pos={0 0}, upload=39772]: Long tail cancellation due to optimal threshold Dec 5 14:09:13.885527824 (45h13m50.356738563s) [noise=true , address=145.79.227.32:4855 , piece_id=N3F2LVAV3UNBWZ7V7HFLNVU5NGWD6X2RB6WKVEVGBJMHUUIJP3PA, seg_pos={0 0}, upload=39772]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TGTRhEi2KsbrkC6ZmgGkMaJri9uhzEcmXe1J4nJjpEd1BE4ka): protocol: write tcp 188.166.225.170:43992->145.79.227.32:4855: use of closed network connection; write tcp 188.166.225.170:43992->145.79.227.32:4855: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43992->145.79.227.32:4855: use of closed network connection Dec 5 14:09:13.885535649 (45h13m50.356746388s) [noise=true , address=145.79.227.32:4855 , piece_id=N3F2LVAV3UNBWZ7V7HFLNVU5NGWD6X2RB6WKVEVGBJMHUUIJP3PA, seg_pos={0 0}, upload=39772]: Long tail cancellation due to optimal threshold Dec 5 14:09:13.885614951 (45h13m50.35682569s) [noise=true , address=64.49.13.169:4705 , piece_id=TTCHF4ENHBWLOJQCAJKMG6BUZGHDDEU3KW5AU5Q3CFCOTKIONE4Q, seg_pos={0 0}, upload=39772]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:44254->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:44254->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44254->64.49.13.169:4705: use of closed network connection Dec 5 14:09:13.885620010 (45h13m50.356830749s) [noise=true , address=64.49.13.169:4705 , piece_id=TTCHF4ENHBWLOJQCAJKMG6BUZGHDDEU3KW5AU5Q3CFCOTKIONE4Q, seg_pos={0 0}, upload=39772]: Long tail cancellation due to optimal threshold Dec 5 14:09:13.885636241 (45h13m50.35684697s) [seg_pos={0 0}, upload=39772]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 5 14:09:13.885731973 (45h13m50.356942722s) [upload=39772]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 5 14:09:13.885739618 (45h13m50.356950357s) [upload=39772]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 5 14:09:14.639752295 (45h13m51.110963024s) [upload=39772]: Done uploading segments... Dec 5 14:09:14.639788193 (45h13m51.110998932s) [upload=39772]: Upload finished. err: <nil> Duration: 10.612693462s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 7.146157798s trace: https://jaeger.dev.storj.io/trace/572c91a59c970da7 FAIL (2025-12-05 14:30:52 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 7.146187024s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 2.767677478s trace: https://jaeger.dev.storj.io/trace/361a2ea17a62744e FAIL (2025-12-05 14:30:49 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 5 14:08:02.649750072 (45h12m39.120960811s) [upload=39762]: Starting upload Dec 5 14:08:02.650012922 (45h12m39.121223681s) [upload=39762]: Done starting upload Dec 5 14:08:02.650140385 (45h12m39.121351124s) [upload=39762]: Uploading segments... Dec 5 14:08:02.650204036 (45h12m39.121414775s) [upload=39762]: Got next segment. Inline: true Dec 5 14:08:02.650210659 (45h12m39.121421398s) [upload=39762]: Waiting for error group managing segments... Dec 5 14:08:02.650285611 (45h12m39.12149635s) [upload=39762]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 5 14:08:02.650292053 (45h12m39.121502792s) [upload=39762]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Dec 5 14:08:02.650295811 (45h12m39.12150655s) [upload=39762]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 5 14:08:05.223576463 (45h12m41.694787212s) [upload=39762]: Done uploading segments... Dec 5 14:08:05.223604205 (45h12m41.694814934s) [upload=39762]: Upload finished. err: <nil> Duration:* 2.767705229s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 45 out of 60 is failed (threshold=6) [XXXXXXXXXXXXX.X..XXXXX..XXXXXX.X.XXXXXXX.XXX.XXXXX.XXX.X....] duration: 5.001872815s trace: https://jaeger.dev.storj.io/trace/75d5f40d5cfca95c FAIL (2025-12-05 14:30:48 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00189154s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 6.070453711s trace: https://jaeger.dev.storj.io/trace/572c91a59c970da7 FAIL (2025-12-05 14:30:34 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 6.070487686s core uplink us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....X....................................X.................] duration: 17.551120931s trace: https://jaeger.dev.storj.io/trace/496d6a85c2db53eb FAIL (2025-12-05 14:30:33 UTC) orbiter warning: context deadline exceeded Duration: 17.551150066s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 23.023135898s trace: https://jaeger.dev.storj.io/trace/65df260343fa2062 FAIL (2025-12-05 14:30:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, Piece ID: 2KY6XA6532G23SRYSNBYNYZ2Y65ZLMXDC2ERGQ6PH4UJQZDUZRUA) context deadline exceeded; piecestore: (Node ID: 12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, Piece ID: JYAT6OFRJQNEZPORFRHUCTFQDPVM2ZTXEUBWGNY6BLADCTZIS66Q) context deadline exceeded; piecestore: (Node ID: 1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, Piece ID: RV4CB7VT7T3YVFFBGXA6EVYMGEOWBU3C3XYUTEMRZDBVHRNVEVLA) context deadline exceeded; piecestore: (Node ID: 12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, Piece ID: VMP5DCNJDH5EAO6LOHSOCP4GGOYI6EV3YHCW5GXB2BRTOO67YQMQ) context deadline exceeded; piecestore: (Node ID: 1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5, Piece ID: UNWONHZ6EIRFROZ5L5YAEFBSMFZHEN4VKAKYM2DWTW4BGCTNMAMA) context deadline exceeded ======LOGS====== Dec 5 13:50:13.914958542 (44h54m50.386169281s) [upload=39497]: Starting upload Dec 5 13:50:13.915042042 (44h54m50.386252771s) [upload=39497]: Done starting upload Dec 5 13:50:13.915237493 (44h54m50.386448232s) [upload=39497]: Uploading segments... Dec 5 13:50:13.915339918 (44h54m50.386550688s) [upload=39497]: Got next segment. Inline: false Dec 5 13:50:13.915395575 (44h54m50.386606314s) [upload=39497]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 5 13:50:13.915400735 (44h54m50.386611464s) [upload=39497]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 5 13:50:14.177967782 (44h54m50.649178531s) [seg_pos={0 0}, upload=39497]: Begin upload segment... Dec 5 13:50:14.178049498 (44h54m50.649260237s) [seg_pos={0 0}, upload=39497]: Done begin upload segment. Dec 5 13:50:14.178089053 (44h54m50.649299792s) [noise=true , address=145.79.227.34:4906 , piece_id=L2SWU22TLNZNS3LO2E3HYCKOVPO6CIOZWOACPQV7CGMURN2HEQRQ, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.178100966 (44h54m50.649311715s) [noise=true , address=145.79.227.10:4402 , piece_id=IHNZYV6LJREQ63GBFJY6BVB54QLSVCZIIUUTLOG7YOOOXNGHQ7WA, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.178155129 (44h54m50.649365868s) [noise=true , address=145.79.227.23:4546 , piece_id=UNWONHZ6EIRFROZ5L5YAEFBSMFZHEN4VKAKYM2DWTW4BGCTNMAMA, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.178737678 (44h54m50.649948417s) [noise=true , address=145.79.227.39:4015 , piece_id=V6WH4XCY2YVP2B3DR4HLI7IFH7MGGPAIBAZHHDBHQQ4NUTCNHP7Q, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.179277977 (44h54m50.650488716s) [noise=true , address=145.79.227.21:4474 , piece_id=J5V4QU72P7D2FGZ7NWO5UT3E7S25CTN4MCSPQMSOWK7R6GK4B4HA, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.179826452 (44h54m50.651037191s) [noise=true , address=145.79.227.21:4475 , piece_id=5IPJJ5RHSXCJPUUXXD62EKMNSWSHPIVEWWYS4X77GE7OT6NCTJUA, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.180516445 (44h54m50.651727174s) [noise=true , address=145.79.227.38:4978 , piece_id=N7OVLKMMMZW75CCZDX5QYOG2SDCO7AKO2FITHFDO3EMDSPJJPAYA, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.181184327 (44h54m50.652395066s) [noise=true , address=145.79.227.43:4157 , piece_id=PFPPMUR22PUIHIPCNXQGN5CYFM4K4OG2EMC5LAMUH26RZKETHZAQ, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.182617817 (44h54m50.653828556s) [noise=true , address=145.79.227.41:4087 , piece_id=2KY6XA6532G23SRYSNBYNYZ2Y65ZLMXDC2ERGQ6PH4UJQZDUZRUA, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.183276120 (44h54m50.654486859s) [noise=true , address=145.79.227.16:4909 , piece_id=JYAT6OFRJQNEZPORFRHUCTFQDPVM2ZTXEUBWGNY6BLADCTZIS66Q, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.184534556 (44h54m50.655745295s) [noise=true , address=145.79.227.20:4438 , piece_id=P64J3PKA4V3SQWAC36KWBVXDBE4KXYJHUXWYHNFZHCQEGQZUWHFQ, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.186480883 (44h54m50.657691622s) [noise=true , address=145.79.227.25:4619 , piece_id=RV4CB7VT7T3YVFFBGXA6EVYMGEOWBU3C3XYUTEMRZDBVHRNVEVLA, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.188448450 (44h54m50.659659189s) [noise=true , address=77.90.31.14:4663 , piece_id=7WW2DVMZSNJTG3J4NANUAIZKMLICAYLP5RAOKKWCTCUTJAISG7SQ, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.190696029 (44h54m50.661906768s) [noise=true , address=145.79.227.4:4187 , piece_id=VMP5DCNJDH5EAO6LOHSOCP4GGOYI6EV3YHCW5GXB2BRTOO67YQMQ, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.193175661 (44h54m50.6643864s) [noise=true , address=145.79.227.31:4799 , piece_id=2C63VGDP7JVQVWSLYLHZEHC4SB7PU6NFUZWLCTEO7AGDP32X3JOQ, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.195649231 (44h54m50.66685997s) [noise=true , address=145.79.227.14:4838 , piece_id=6EVD7RSXQJOFT645D7ILWLUWUBYFP6IRR5GESYOAWXF2IXKJEUCA, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:15.979218426 (44h54m52.450429165s) [upload=39497]: Next returned nil segment Dec 5 13:50:15.979231121 (44h54m52.45044185s) [upload=39497]: Waiting for error group managing segments... Dec 5 13:50:16.323526524 (44h54m52.794737263s) [noise=true , address=145.79.227.21:4475 , piece_id=5IPJJ5RHSXCJPUUXXD62EKMNSWSHPIVEWWYS4X77GE7OT6NCTJUA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:16.347685978 (44h54m52.818896717s) [noise=true , address=145.79.227.23:4546 , piece_id=UNWONHZ6EIRFROZ5L5YAEFBSMFZHEN4VKAKYM2DWTW4BGCTNMAMA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:16.351351307 (44h54m52.822562036s) [noise=true , address=145.79.227.10:4402 , piece_id=IHNZYV6LJREQ63GBFJY6BVB54QLSVCZIIUUTLOG7YOOOXNGHQ7WA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:16.351375874 (44h54m52.822586603s) [noise=true , address=145.79.227.38:4978 , piece_id=N7OVLKMMMZW75CCZDX5QYOG2SDCO7AKO2FITHFDO3EMDSPJJPAYA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:16.359501778 (44h54m52.830712517s) [noise=true , address=145.79.227.31:4799 , piece_id=2C63VGDP7JVQVWSLYLHZEHC4SB7PU6NFUZWLCTEO7AGDP32X3JOQ, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:16.360729425 (44h54m52.831940164s) [noise=true , address=145.79.227.4:4187 , piece_id=VMP5DCNJDH5EAO6LOHSOCP4GGOYI6EV3YHCW5GXB2BRTOO67YQMQ, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:16.371054476 (44h54m52.842265225s) [noise=true , address=145.79.227.41:4087 , piece_id=2KY6XA6532G23SRYSNBYNYZ2Y65ZLMXDC2ERGQ6PH4UJQZDUZRUA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:16.392350999 (44h54m52.863561738s) [noise=true , address=77.90.31.14:4663 , piece_id=7WW2DVMZSNJTG3J4NANUAIZKMLICAYLP5RAOKKWCTCUTJAISG7SQ, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:16.437887268 (44h54m52.909098007s) [noise=true , address=145.79.227.21:4474 , piece_id=J5V4QU72P7D2FGZ7NWO5UT3E7S25CTN4MCSPQMSOWK7R6GK4B4HA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:16.642787347 (44h54m53.113998086s) [noise=true , address=145.79.227.43:4157 , piece_id=PFPPMUR22PUIHIPCNXQGN5CYFM4K4OG2EMC5LAMUH26RZKETHZAQ, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:16.987804677 (44h54m53.459015416s) [noise=true , address=145.79.227.25:4619 , piece_id=RV4CB7VT7T3YVFFBGXA6EVYMGEOWBU3C3XYUTEMRZDBVHRNVEVLA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:17.014774723 (44h54m53.485985462s) [noise=true , address=145.79.227.14:4838 , piece_id=6EVD7RSXQJOFT645D7ILWLUWUBYFP6IRR5GESYOAWXF2IXKJEUCA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:17.413923466 (44h54m53.885134205s) [noise=true , address=145.79.227.16:4909 , piece_id=JYAT6OFRJQNEZPORFRHUCTFQDPVM2ZTXEUBWGNY6BLADCTZIS66Q, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:17.413981627 (44h54m53.885192367s) [seg_pos={0 0}, upload=39497]: Segment reached optimal threshold of 13 pieces. Dec 5 13:50:17.414271349 (44h54m53.885482088s) [noise=true , address=145.79.227.39:4015 , piece_id=V6WH4XCY2YVP2B3DR4HLI7IFH7MGGPAIBAZHHDBHQQ4NUTCNHP7Q, seg_pos={0 0}, upload=39497]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE): protocol: write tcp 188.166.225.170:56230->145.79.227.39:4015: use of closed network connection; write tcp 188.166.225.170:56230->145.79.227.39:4015: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56230->145.79.227.39:4015: use of closed network connection Dec 5 13:50:17.414282390 (44h54m53.885493119s) [noise=true , address=145.79.227.39:4015 , piece_id=V6WH4XCY2YVP2B3DR4HLI7IFH7MGGPAIBAZHHDBHQQ4NUTCNHP7Q, seg_pos={0 0}, upload=39497]: Long tail cancellation due to optimal threshold Dec 5 13:50:17.414355008 (44h54m53.885565747s) [noise=true , address=145.79.227.34:4906 , piece_id=L2SWU22TLNZNS3LO2E3HYCKOVPO6CIOZWOACPQV7CGMURN2HEQRQ, seg_pos={0 0}, upload=39497]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 5 13:50:17.414365318 (44h54m53.885576057s) [noise=true , address=145.79.227.34:4906 , piece_id=L2SWU22TLNZNS3LO2E3HYCKOVPO6CIOZWOACPQV7CGMURN2HEQRQ, seg_pos={0 0}, upload=39497]: Long tail cancellation due to optimal threshold Dec 5 13:50:17.414811999 (44h54m53.886022758s) [noise=true , address=145.79.227.20:4438 , piece_id=P64J3PKA4V3SQWAC36KWBVXDBE4KXYJHUXWYHNFZHCQEGQZUWHFQ, seg_pos={0 0}, upload=39497]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT): protocol: expected piece hash; context canceled; EOF Dec 5 13:50:17.414824472 (44h54m53.886035211s) [noise=true , address=145.79.227.20:4438 , piece_id=P64J3PKA4V3SQWAC36KWBVXDBE4KXYJHUXWYHNFZHCQEGQZUWHFQ, seg_pos={0 0}, upload=39497]: Long tail cancellation due to optimal threshold Dec 5 13:50:17.414850933 (44h54m53.886061672s) [seg_pos={0 0}, upload=39497]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 5 13:50:17.414965771 (44h54m53.88617652s) [upload=39497]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 5 13:50:17.414973846 (44h54m53.886184585s) [upload=39497]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 5 13:50:17.720870703 (44h54m54.192081452s) [upload=39497]: Done uploading segments... Dec 5 13:50:17.720894969 (44h54m54.192105708s) [upload=39497]: Upload finished. err: <nil> Duration: 23.023170194s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 2.969191848s trace: https://jaeger.dev.storj.io/trace/41b9575307efc1e3 FAIL (2025-12-05 14:30:07 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 2.969224991s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 9.10460667s trace: https://jaeger.dev.storj.io/trace/70e54e58d52fa95e FAIL (2025-12-05 14:30:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1MntjJtcCDSRozKpesp6S2ZjYrMhxiQheDiPMPSCnavWBBGYt3, Piece ID: MQGKTFSYGQSOMNUEIVOGGS5ZEVM52CLW4TEG7DA5ICOK73GEAI5Q) context deadline exceeded; piecestore: (Node ID: 12YnWetw1NereHQi7cwHCnmu1LDjRFNpB2MQHSWQrjQdN1CZbMw, Piece ID: UALY4LTNPIS62L46OIH4PTKRES72I7LA55UL7B2UCZLRT7DQ7RVQ) context deadline exceeded; piecestore: (Node ID: 12Dfm6iHQbMhj1YWLu3UboTCtZ2hq87aiYUgH7Kig3pDUrPeFj4, Piece ID: 4W72IA43JPPLQ5ZDZHZDW6AFEZDVG7AHJT7555P4BY2Z3QYMLKNA) context deadline exceeded; piecestore: (Node ID: 12Qb6sMhCyzYd4S15Td1kzD7kgqNKSi2avG15CQRSG8bcDVvGzi, Piece ID: SUX7UBSIQIIYU327MGSXKLJBPXFW6UEVI3NJRAWVEZ3GBUN2QCJQ) context deadline exceeded; piecestore: (Node ID: 1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks, Piece ID: WBCLEOB7XBAT2MPT6M7NLLNQM2JR3S22BV6UZBSGRDFIJP5JS2LA) context deadline exceeded; piecestore: (Node ID: 123QEF1LQm8nDCTjZvq51BjSMTYwHjhj2ajZd7bCRjiWJVLGXyS, Piece ID: MBHG2UWXQMEZPJJHW67KJMJTIVT4QWPXKDIRBRVTBAFN46J3M42Q) context deadline exceeded; piecestore: (Node ID: 1gPa2A8V3hY6WFgDTMk5ZMQpEaVVuhjXFQYHoZhK2R9JUPyepz, Piece ID: ZS3JXULFI4AGS3HIIX2POYQ3XLBG6GPFAWBAV2NBRLMO447DH7ZA) context deadline exceeded; piecestore: (Node ID: 121RT7tP6czMFAiZ3DjzSSsYgPt3xZXVbXVpkcYxaZYscNiMjL6, Piece ID: EPHUILAEI77HCSYICLO2IMEGVISUXUEQGKNAHV52VDAGDCAP3RKQ) context deadline exceeded; piecestore: (Node ID: 1if7vf1cNTBwAhb6owi6tuGLYqSfa3feWhsvGSevwuwykhz2RY, Piece ID: CFMB4CDU7DWOF3L5T7DASMJB33RVAYSTMM4WFNUSUNV4H6CGPNXQ) context deadline exceeded; piecestore: (Node ID: 1ArQdwHzHSm7c2QZkTt1kQsfzocvCNG2D9yoQgnmbS7HYLCXhy, Piece ID: IY7G4GQVUEYHNO4HJAD3CHGX4SQQNGWPMUXUJJKLC7MUTUUFGFWA) context deadline exceeded ======LOGS====== 4909 , piece_id=EPHUILAEI77HCSYICLO2IMEGVISUXUEQGKNAHV52VDAGDCAP3RKQ, seg_pos={0 0}, upload=39772]: Uploading piece... Dec 5 14:09:11.669180895 (45h13m48.140391634s) [noise=true , address=77.90.31.12:4616 , piece_id=SUX7UBSIQIIYU327MGSXKLJBPXFW6UEVI3NJRAWVEZ3GBUN2QCJQ, seg_pos={0 0}, upload=39772]: Uploading piece... Dec 5 14:09:11.669641381 (45h13m48.14085212s) [noise=true , address=145.79.227.32:4855 , piece_id=N3F2LVAV3UNBWZ7V7HFLNVU5NGWD6X2RB6WKVEVGBJMHUUIJP3PA, seg_pos={0 0}, upload=39772]: Uploading piece... Dec 5 14:09:11.670111447 (45h13m48.141322196s) [noise=true , address=77.90.31.17:4794 , piece_id=ALZXJQTN64GGUOZPJMGJ6CRXQBO2R5PRVNUBQHRKWYGY7KJF6CVQ, seg_pos={0 0}, upload=39772]: Uploading piece... Dec 5 14:09:11.670601259 (45h13m48.141811999s) [noise=true , address=64.49.13.169:4705 , piece_id=TTCHF4ENHBWLOJQCAJKMG6BUZGHDDEU3KW5AU5Q3CFCOTKIONE4Q, seg_pos={0 0}, upload=39772]: Uploading piece... Dec 5 14:09:11.671075502 (45h13m48.142286241s) [noise=true , address=77.90.31.13:4654 , piece_id=CEAQAXPEBEPYGOMBFNVRRYRCT67SMMNEVDBADWIOPQXEJCN3EGZQ, seg_pos={0 0}, upload=39772]: Uploading piece... Dec 5 14:09:11.671598037 (45h13m48.142808776s) [noise=true , address=45.91.116.11:4147 , piece_id=UALY4LTNPIS62L46OIH4PTKRES72I7LA55UL7B2UCZLRT7DQ7RVQ, seg_pos={0 0}, upload=39772]: Uploading piece... Dec 5 14:09:11.672127346 (45h13m48.143338105s) [noise=true , address=145.79.227.15:4907 , piece_id=BXEPFHK6AENJZM2APP5LPV2LHJ4L7CBFQF5JSQJQBC6F5AKDGYQA, seg_pos={0 0}, upload=39772]: Uploading piece... Dec 5 14:09:11.672648588 (45h13m48.143859327s) [noise=true , address=77.90.31.18:4838 , piece_id=CFMB4CDU7DWOF3L5T7DASMJB33RVAYSTMM4WFNUSUNV4H6CGPNXQ, seg_pos={0 0}, upload=39772]: Uploading piece... Dec 5 14:09:11.673155192 (45h13m48.144365931s) [noise=true , address=77.90.31.19:4856 , piece_id=MQGKTFSYGQSOMNUEIVOGGS5ZEVM52CLW4TEG7DA5ICOK73GEAI5Q, seg_pos={0 0}, upload=39772]: Uploading piece... Dec 5 14:09:11.673691684 (45h13m48.144902423s) [noise=true , address=45.91.116.10:4091 , piece_id=ZS3JXULFI4AGS3HIIX2POYQ3XLBG6GPFAWBAV2NBRLMO447DH7ZA, seg_pos={0 0}, upload=39772]: Uploading piece... Dec 5 14:09:11.674805966 (45h13m48.146016705s) [noise=true , address=145.79.227.19:4097 , piece_id=MKAKS25RPIEVHT2WJJ7QYY4FBS7JCIM3RKY7MWRJ7FZTZ2YZXMGQ, seg_pos={0 0}, upload=39772]: Uploading piece... Dec 5 14:09:11.675360772 (45h13m48.146571511s) [noise=true , address=45.91.116.5:4329 , piece_id=MBHG2UWXQMEZPJJHW67KJMJTIVT4QWPXKDIRBRVTBAFN46J3M42Q, seg_pos={0 0}, upload=39772]: Uploading piece... Dec 5 14:09:13.239845356 (45h13m49.711056085s) [upload=39772]: Next returned nil segment Dec 5 14:09:13.239860604 (45h13m49.711071334s) [upload=39772]: Waiting for error group managing segments... Dec 5 14:09:13.643777330 (45h13m50.114988069s) [noise=true , address=77.90.31.13:4654 , piece_id=CEAQAXPEBEPYGOMBFNVRRYRCT67SMMNEVDBADWIOPQXEJCN3EGZQ, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil> Dec 5 14:09:13.697999800 (45h13m50.169210539s) [noise=true , address=77.90.31.18:4838 , piece_id=CFMB4CDU7DWOF3L5T7DASMJB33RVAYSTMM4WFNUSUNV4H6CGPNXQ, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil> Dec 5 14:09:13.713127946 (45h13m50.184338685s) [noise=true , address=77.90.31.15:4729 , piece_id=IY7G4GQVUEYHNO4HJAD3CHGX4SQQNGWPMUXUJJKLC7MUTUUFGFWA, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil> Dec 5 14:09:13.718498752 (45h13m50.189709491s) [noise=true , address=77.90.31.21:4909 , piece_id=EPHUILAEI77HCSYICLO2IMEGVISUXUEQGKNAHV52VDAGDCAP3RKQ, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil> Dec 5 14:09:13.726522451 (45h13m50.19773318s) [noise=true , address=77.90.31.17:4794 , piece_id=ALZXJQTN64GGUOZPJMGJ6CRXQBO2R5PRVNUBQHRKWYGY7KJF6CVQ, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil> Dec 5 14:09:13.749057442 (45h13m50.220268171s) [noise=true , address=45.91.116.10:4091 , piece_id=ZS3JXULFI4AGS3HIIX2POYQ3XLBG6GPFAWBAV2NBRLMO447DH7ZA, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil> Dec 5 14:09:13.750808776 (45h13m50.222019515s) [noise=true , address=77.90.31.10:4542 , piece_id=XDD4QZAKCWJK5A3F5FDL4NRNPEQZF3GKODCGVT2V47AH6KVEOOWQ, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil> Dec 5 14:09:13.754762756 (45h13m50.225973485s) [noise=true , address=77.90.31.11:4556 , piece_id=WBCLEOB7XBAT2MPT6M7NLLNQM2JR3S22BV6UZBSGRDFIJP5JS2LA, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil> Dec 5 14:09:13.759242185 (45h13m50.230452925s) [noise=true , address=77.90.31.12:4616 , piece_id=SUX7UBSIQIIYU327MGSXKLJBPXFW6UEVI3NJRAWVEZ3GBUN2QCJQ, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil> Dec 5 14:09:13.774028982 (45h13m50.245239711s) [noise=true , address=77.90.31.19:4856 , piece_id=MQGKTFSYGQSOMNUEIVOGGS5ZEVM52CLW4TEG7DA5ICOK73GEAI5Q, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil> Dec 5 14:09:13.786684680 (45h13m50.257895419s) [noise=true , address=45.91.116.9:4272 , piece_id=4W72IA43JPPLQ5ZDZHZDW6AFEZDVG7AHJT7555P4BY2Z3QYMLKNA, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil> Dec 5 14:09:13.790669208 (45h13m50.261879947s) [noise=true , address=45.91.116.11:4147 , piece_id=UALY4LTNPIS62L46OIH4PTKRES72I7LA55UL7B2UCZLRT7DQ7RVQ, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil> Dec 5 14:09:13.850747877 (45h13m50.321958617s) [noise=true , address=45.91.116.8:4071 , piece_id=3YI67CEXFW75VRROYLSABR55DNVFHE4SR6EUP57CPRIIB63GZOVA, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil> Dec 5 14:09:13.884817071 (45h13m50.3560278s) [noise=true , address=45.91.116.5:4329 , piece_id=MBHG2UWXQMEZPJJHW67KJMJTIVT4QWPXKDIRBRVTBAFN46J3M42Q, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil> Dec 5 14:09:13.884838361 (45h13m50.35604909s) [seg_pos={0 0}, upload=39772]: Segment reached optimal threshold of 14 pieces. Dec 5 14:09:13.885054584 (45h13m50.356265313s) [noise=true , address=145.79.227.15:4907 , piece_id=BXEPFHK6AENJZM2APP5LPV2LHJ4L7CBFQF5JSQJQBC6F5AKDGYQA, seg_pos={0 0}, upload=39772]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JNee734A635XWMM3Q3N6siDsvKY3FEMQerDjYZrQR6orGBaY3): protocol: write tcp 188.166.225.170:59612->145.79.227.15:4907: use of closed network connection; write tcp 188.166.225.170:59612->145.79.227.15:4907: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59612->145.79.227.15:4907: use of closed network connection Dec 5 14:09:13.885064302 (45h13m50.356275041s) [noise=true , address=145.79.227.15:4907 , piece_id=BXEPFHK6AENJZM2APP5LPV2LHJ4L7CBFQF5JSQJQBC6F5AKDGYQA, seg_pos={0 0}, upload=39772]: Long tail cancellation due to optimal threshold Dec 5 14:09:13.885416693 (45h13m50.356627432s) [noise=true , address=145.79.227.19:4097 , piece_id=MKAKS25RPIEVHT2WJJ7QYY4FBS7JCIM3RKY7MWRJ7FZTZ2YZXMGQ, seg_pos={0 0}, upload=39772]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QP2aZ9BC22cD1Nzz2E12DiA1XoNu3GGTgHC8Cc3TvpQM3Ubn3): protocol: write tcp 188.166.225.170:37344->145.79.227.19:4097: use of closed network connection; write tcp 188.166.225.170:37344->145.79.227.19:4097: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37344->145.79.227.19:4097: use of closed network connection Dec 5 14:09:13.885427744 (45h13m50.356638483s) [noise=true , address=145.79.227.19:4097 , piece_id=MKAKS25RPIEVHT2WJJ7QYY4FBS7JCIM3RKY7MWRJ7FZTZ2YZXMGQ, seg_pos={0 0}, upload=39772]: Long tail cancellation due to optimal threshold Dec 5 14:09:13.885527824 (45h13m50.356738563s) [noise=true , address=145.79.227.32:4855 , piece_id=N3F2LVAV3UNBWZ7V7HFLNVU5NGWD6X2RB6WKVEVGBJMHUUIJP3PA, seg_pos={0 0}, upload=39772]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TGTRhEi2KsbrkC6ZmgGkMaJri9uhzEcmXe1J4nJjpEd1BE4ka): protocol: write tcp 188.166.225.170:43992->145.79.227.32:4855: use of closed network connection; write tcp 188.166.225.170:43992->145.79.227.32:4855: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43992->145.79.227.32:4855: use of closed network connection Dec 5 14:09:13.885535649 (45h13m50.356746388s) [noise=true , address=145.79.227.32:4855 , piece_id=N3F2LVAV3UNBWZ7V7HFLNVU5NGWD6X2RB6WKVEVGBJMHUUIJP3PA, seg_pos={0 0}, upload=39772]: Long tail cancellation due to optimal threshold Dec 5 14:09:13.885614951 (45h13m50.35682569s) [noise=true , address=64.49.13.169:4705 , piece_id=TTCHF4ENHBWLOJQCAJKMG6BUZGHDDEU3KW5AU5Q3CFCOTKIONE4Q, seg_pos={0 0}, upload=39772]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:44254->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:44254->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44254->64.49.13.169:4705: use of closed network connection Dec 5 14:09:13.885620010 (45h13m50.356830749s) [noise=true , address=64.49.13.169:4705 , piece_id=TTCHF4ENHBWLOJQCAJKMG6BUZGHDDEU3KW5AU5Q3CFCOTKIONE4Q, seg_pos={0 0}, upload=39772]: Long tail cancellation due to optimal threshold Dec 5 14:09:13.885636241 (45h13m50.35684697s) [seg_pos={0 0}, upload=39772]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 5 14:09:13.885731973 (45h13m50.356942722s) [upload=39772]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 5 14:09:13.885739618 (45h13m50.356950357s) [upload=39772]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 5 14:09:14.639752295 (45h13m51.110963024s) [upload=39772]: Done uploading segments... Dec 5 14:09:14.639788193 (45h13m51.110998932s) [upload=39772]: Upload finished. err: <nil> Duration: 9.104635394s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......X....................................X................] duration: 14.070753891s trace: https://jaeger.dev.storj.io/trace/496d6a85c2db53eb FAIL (2025-12-05 14:29:49 UTC) orbiter warning: context deadline exceeded Duration: 14.070817652s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 7.518521997s trace: https://jaeger.dev.storj.io/trace/572c91a59c970da7 FAIL (2025-12-05 14:29:47 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 7.518548598s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 4.074083532s trace: https://jaeger.dev.storj.io/trace/41b9575307efc1e3 FAIL (2025-12-05 14:29:44 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 4.074101828s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 2.831781992s trace: https://jaeger.dev.storj.io/trace/361a2ea17a62744e FAIL (2025-12-05 14:29:43 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 5 14:08:02.649750072 (45h12m39.120960811s) [upload=39762]: Starting upload Dec 5 14:08:02.650012922 (45h12m39.121223681s) [upload=39762]: Done starting upload Dec 5 14:08:02.650140385 (45h12m39.121351124s) [upload=39762]: Uploading segments... Dec 5 14:08:02.650204036 (45h12m39.121414775s) [upload=39762]: Got next segment. Inline: true Dec 5 14:08:02.650210659 (45h12m39.121421398s) [upload=39762]: Waiting for error group managing segments... Dec 5 14:08:02.650285611 (45h12m39.12149635s) [upload=39762]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 5 14:08:02.650292053 (45h12m39.121502792s) [upload=39762]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Dec 5 14:08:02.650295811 (45h12m39.12150655s) [upload=39762]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 5 14:08:05.223576463 (45h12m41.694787212s) [upload=39762]: Done uploading segments... Dec 5 14:08:05.223604205 (45h12m41.694814934s) [upload=39762]: Upload finished. err: <nil> Duration:* 2.831807731s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 15.417734841s trace: https://jaeger.dev.storj.io/trace/65df260343fa2062 FAIL (2025-12-05 14:29:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, Piece ID: 2KY6XA6532G23SRYSNBYNYZ2Y65ZLMXDC2ERGQ6PH4UJQZDUZRUA) context deadline exceeded; piecestore: (Node ID: 12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, Piece ID: JYAT6OFRJQNEZPORFRHUCTFQDPVM2ZTXEUBWGNY6BLADCTZIS66Q) context deadline exceeded; piecestore: (Node ID: 1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, Piece ID: RV4CB7VT7T3YVFFBGXA6EVYMGEOWBU3C3XYUTEMRZDBVHRNVEVLA) context deadline exceeded; piecestore: (Node ID: 12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, Piece ID: VMP5DCNJDH5EAO6LOHSOCP4GGOYI6EV3YHCW5GXB2BRTOO67YQMQ) context deadline exceeded; piecestore: (Node ID: 1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5, Piece ID: UNWONHZ6EIRFROZ5L5YAEFBSMFZHEN4VKAKYM2DWTW4BGCTNMAMA) context deadline exceeded ======LOGS====== Dec 5 13:50:13.914958542 (44h54m50.386169281s) [upload=39497]: Starting upload Dec 5 13:50:13.915042042 (44h54m50.386252771s) [upload=39497]: Done starting upload Dec 5 13:50:13.915237493 (44h54m50.386448232s) [upload=39497]: Uploading segments... Dec 5 13:50:13.915339918 (44h54m50.386550688s) [upload=39497]: Got next segment. Inline: false Dec 5 13:50:13.915395575 (44h54m50.386606314s) [upload=39497]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 5 13:50:13.915400735 (44h54m50.386611464s) [upload=39497]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 5 13:50:14.177967782 (44h54m50.649178531s) [seg_pos={0 0}, upload=39497]: Begin upload segment... Dec 5 13:50:14.178049498 (44h54m50.649260237s) [seg_pos={0 0}, upload=39497]: Done begin upload segment. Dec 5 13:50:14.178089053 (44h54m50.649299792s) [noise=true , address=145.79.227.34:4906 , piece_id=L2SWU22TLNZNS3LO2E3HYCKOVPO6CIOZWOACPQV7CGMURN2HEQRQ, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.178100966 (44h54m50.649311715s) [noise=true , address=145.79.227.10:4402 , piece_id=IHNZYV6LJREQ63GBFJY6BVB54QLSVCZIIUUTLOG7YOOOXNGHQ7WA, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.178155129 (44h54m50.649365868s) [noise=true , address=145.79.227.23:4546 , piece_id=UNWONHZ6EIRFROZ5L5YAEFBSMFZHEN4VKAKYM2DWTW4BGCTNMAMA, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.178737678 (44h54m50.649948417s) [noise=true , address=145.79.227.39:4015 , piece_id=V6WH4XCY2YVP2B3DR4HLI7IFH7MGGPAIBAZHHDBHQQ4NUTCNHP7Q, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.179277977 (44h54m50.650488716s) [noise=true , address=145.79.227.21:4474 , piece_id=J5V4QU72P7D2FGZ7NWO5UT3E7S25CTN4MCSPQMSOWK7R6GK4B4HA, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.179826452 (44h54m50.651037191s) [noise=true , address=145.79.227.21:4475 , piece_id=5IPJJ5RHSXCJPUUXXD62EKMNSWSHPIVEWWYS4X77GE7OT6NCTJUA, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.180516445 (44h54m50.651727174s) [noise=true , address=145.79.227.38:4978 , piece_id=N7OVLKMMMZW75CCZDX5QYOG2SDCO7AKO2FITHFDO3EMDSPJJPAYA, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.181184327 (44h54m50.652395066s) [noise=true , address=145.79.227.43:4157 , piece_id=PFPPMUR22PUIHIPCNXQGN5CYFM4K4OG2EMC5LAMUH26RZKETHZAQ, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.182617817 (44h54m50.653828556s) [noise=true , address=145.79.227.41:4087 , piece_id=2KY6XA6532G23SRYSNBYNYZ2Y65ZLMXDC2ERGQ6PH4UJQZDUZRUA, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.183276120 (44h54m50.654486859s) [noise=true , address=145.79.227.16:4909 , piece_id=JYAT6OFRJQNEZPORFRHUCTFQDPVM2ZTXEUBWGNY6BLADCTZIS66Q, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.184534556 (44h54m50.655745295s) [noise=true , address=145.79.227.20:4438 , piece_id=P64J3PKA4V3SQWAC36KWBVXDBE4KXYJHUXWYHNFZHCQEGQZUWHFQ, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.186480883 (44h54m50.657691622s) [noise=true , address=145.79.227.25:4619 , piece_id=RV4CB7VT7T3YVFFBGXA6EVYMGEOWBU3C3XYUTEMRZDBVHRNVEVLA, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.188448450 (44h54m50.659659189s) [noise=true , address=77.90.31.14:4663 , piece_id=7WW2DVMZSNJTG3J4NANUAIZKMLICAYLP5RAOKKWCTCUTJAISG7SQ, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.190696029 (44h54m50.661906768s) [noise=true , address=145.79.227.4:4187 , piece_id=VMP5DCNJDH5EAO6LOHSOCP4GGOYI6EV3YHCW5GXB2BRTOO67YQMQ, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.193175661 (44h54m50.6643864s) [noise=true , address=145.79.227.31:4799 , piece_id=2C63VGDP7JVQVWSLYLHZEHC4SB7PU6NFUZWLCTEO7AGDP32X3JOQ, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.195649231 (44h54m50.66685997s) [noise=true , address=145.79.227.14:4838 , piece_id=6EVD7RSXQJOFT645D7ILWLUWUBYFP6IRR5GESYOAWXF2IXKJEUCA, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:15.979218426 (44h54m52.450429165s) [upload=39497]: Next returned nil segment Dec 5 13:50:15.979231121 (44h54m52.45044185s) [upload=39497]: Waiting for error group managing segments... Dec 5 13:50:16.323526524 (44h54m52.794737263s) [noise=true , address=145.79.227.21:4475 , piece_id=5IPJJ5RHSXCJPUUXXD62EKMNSWSHPIVEWWYS4X77GE7OT6NCTJUA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:16.347685978 (44h54m52.818896717s) [noise=true , address=145.79.227.23:4546 , piece_id=UNWONHZ6EIRFROZ5L5YAEFBSMFZHEN4VKAKYM2DWTW4BGCTNMAMA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:16.351351307 (44h54m52.822562036s) [noise=true , address=145.79.227.10:4402 , piece_id=IHNZYV6LJREQ63GBFJY6BVB54QLSVCZIIUUTLOG7YOOOXNGHQ7WA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:16.351375874 (44h54m52.822586603s) [noise=true , address=145.79.227.38:4978 , piece_id=N7OVLKMMMZW75CCZDX5QYOG2SDCO7AKO2FITHFDO3EMDSPJJPAYA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:16.359501778 (44h54m52.830712517s) [noise=true , address=145.79.227.31:4799 , piece_id=2C63VGDP7JVQVWSLYLHZEHC4SB7PU6NFUZWLCTEO7AGDP32X3JOQ, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:16.360729425 (44h54m52.831940164s) [noise=true , address=145.79.227.4:4187 , piece_id=VMP5DCNJDH5EAO6LOHSOCP4GGOYI6EV3YHCW5GXB2BRTOO67YQMQ, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:16.371054476 (44h54m52.842265225s) [noise=true , address=145.79.227.41:4087 , piece_id=2KY6XA6532G23SRYSNBYNYZ2Y65ZLMXDC2ERGQ6PH4UJQZDUZRUA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:16.392350999 (44h54m52.863561738s) [noise=true , address=77.90.31.14:4663 , piece_id=7WW2DVMZSNJTG3J4NANUAIZKMLICAYLP5RAOKKWCTCUTJAISG7SQ, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:16.437887268 (44h54m52.909098007s) [noise=true , address=145.79.227.21:4474 , piece_id=J5V4QU72P7D2FGZ7NWO5UT3E7S25CTN4MCSPQMSOWK7R6GK4B4HA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:16.642787347 (44h54m53.113998086s) [noise=true , address=145.79.227.43:4157 , piece_id=PFPPMUR22PUIHIPCNXQGN5CYFM4K4OG2EMC5LAMUH26RZKETHZAQ, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:16.987804677 (44h54m53.459015416s) [noise=true , address=145.79.227.25:4619 , piece_id=RV4CB7VT7T3YVFFBGXA6EVYMGEOWBU3C3XYUTEMRZDBVHRNVEVLA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:17.014774723 (44h54m53.485985462s) [noise=true , address=145.79.227.14:4838 , piece_id=6EVD7RSXQJOFT645D7ILWLUWUBYFP6IRR5GESYOAWXF2IXKJEUCA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:17.413923466 (44h54m53.885134205s) [noise=true , address=145.79.227.16:4909 , piece_id=JYAT6OFRJQNEZPORFRHUCTFQDPVM2ZTXEUBWGNY6BLADCTZIS66Q, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:17.413981627 (44h54m53.885192367s) [seg_pos={0 0}, upload=39497]: Segment reached optimal threshold of 13 pieces. Dec 5 13:50:17.414271349 (44h54m53.885482088s) [noise=true , address=145.79.227.39:4015 , piece_id=V6WH4XCY2YVP2B3DR4HLI7IFH7MGGPAIBAZHHDBHQQ4NUTCNHP7Q, seg_pos={0 0}, upload=39497]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE): protocol: write tcp 188.166.225.170:56230->145.79.227.39:4015: use of closed network connection; write tcp 188.166.225.170:56230->145.79.227.39:4015: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56230->145.79.227.39:4015: use of closed network connection Dec 5 13:50:17.414282390 (44h54m53.885493119s) [noise=true , address=145.79.227.39:4015 , piece_id=V6WH4XCY2YVP2B3DR4HLI7IFH7MGGPAIBAZHHDBHQQ4NUTCNHP7Q, seg_pos={0 0}, upload=39497]: Long tail cancellation due to optimal threshold Dec 5 13:50:17.414355008 (44h54m53.885565747s) [noise=true , address=145.79.227.34:4906 , piece_id=L2SWU22TLNZNS3LO2E3HYCKOVPO6CIOZWOACPQV7CGMURN2HEQRQ, seg_pos={0 0}, upload=39497]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 5 13:50:17.414365318 (44h54m53.885576057s) [noise=true , address=145.79.227.34:4906 , piece_id=L2SWU22TLNZNS3LO2E3HYCKOVPO6CIOZWOACPQV7CGMURN2HEQRQ, seg_pos={0 0}, upload=39497]: Long tail cancellation due to optimal threshold Dec 5 13:50:17.414811999 (44h54m53.886022758s) [noise=true , address=145.79.227.20:4438 , piece_id=P64J3PKA4V3SQWAC36KWBVXDBE4KXYJHUXWYHNFZHCQEGQZUWHFQ, seg_pos={0 0}, upload=39497]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT): protocol: expected piece hash; context canceled; EOF Dec 5 13:50:17.414824472 (44h54m53.886035211s) [noise=true , address=145.79.227.20:4438 , piece_id=P64J3PKA4V3SQWAC36KWBVXDBE4KXYJHUXWYHNFZHCQEGQZUWHFQ, seg_pos={0 0}, upload=39497]: Long tail cancellation due to optimal threshold Dec 5 13:50:17.414850933 (44h54m53.886061672s) [seg_pos={0 0}, upload=39497]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 5 13:50:17.414965771 (44h54m53.88617652s) [upload=39497]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 5 13:50:17.414973846 (44h54m53.886184585s) [upload=39497]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 5 13:50:17.720870703 (44h54m54.192081452s) [upload=39497]: Done uploading segments... Dec 5 13:50:17.720894969 (44h54m54.192105708s) [upload=39497]: Upload finished. err: <nil> Duration: 15.417758536s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 46 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXX.X..XXXXX..XXXXXX.X.XXXXXXX.XXX.XXXXX.XXX.X...] duration: 5.000992459s trace: https://jaeger.dev.storj.io/trace/75d5f40d5cfca95c FAIL (2025-12-05 14:29:36 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001027976s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......X....................................X...............] duration: 15.205739875s trace: https://jaeger.dev.storj.io/trace/496d6a85c2db53eb FAIL (2025-12-05 14:29:18 UTC) orbiter warning: context deadline exceeded Duration: 15.205765973s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 8.961682551s trace: https://jaeger.dev.storj.io/trace/70e54e58d52fa95e FAIL (2025-12-05 14:29:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1MntjJtcCDSRozKpesp6S2ZjYrMhxiQheDiPMPSCnavWBBGYt3, Piece ID: MQGKTFSYGQSOMNUEIVOGGS5ZEVM52CLW4TEG7DA5ICOK73GEAI5Q) context deadline exceeded; piecestore: (Node ID: 12YnWetw1NereHQi7cwHCnmu1LDjRFNpB2MQHSWQrjQdN1CZbMw, Piece ID: UALY4LTNPIS62L46OIH4PTKRES72I7LA55UL7B2UCZLRT7DQ7RVQ) context deadline exceeded; piecestore: (Node ID: 12Dfm6iHQbMhj1YWLu3UboTCtZ2hq87aiYUgH7Kig3pDUrPeFj4, Piece ID: 4W72IA43JPPLQ5ZDZHZDW6AFEZDVG7AHJT7555P4BY2Z3QYMLKNA) context deadline exceeded; piecestore: (Node ID: 12Qb6sMhCyzYd4S15Td1kzD7kgqNKSi2avG15CQRSG8bcDVvGzi, Piece ID: SUX7UBSIQIIYU327MGSXKLJBPXFW6UEVI3NJRAWVEZ3GBUN2QCJQ) context deadline exceeded; piecestore: (Node ID: 1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks, Piece ID: WBCLEOB7XBAT2MPT6M7NLLNQM2JR3S22BV6UZBSGRDFIJP5JS2LA) context deadline exceeded; piecestore: (Node ID: 123QEF1LQm8nDCTjZvq51BjSMTYwHjhj2ajZd7bCRjiWJVLGXyS, Piece ID: MBHG2UWXQMEZPJJHW67KJMJTIVT4QWPXKDIRBRVTBAFN46J3M42Q) context deadline exceeded; piecestore: (Node ID: 1gPa2A8V3hY6WFgDTMk5ZMQpEaVVuhjXFQYHoZhK2R9JUPyepz, Piece ID: ZS3JXULFI4AGS3HIIX2POYQ3XLBG6GPFAWBAV2NBRLMO447DH7ZA) context deadline exceeded; piecestore: (Node ID: 121RT7tP6czMFAiZ3DjzSSsYgPt3xZXVbXVpkcYxaZYscNiMjL6, Piece ID: EPHUILAEI77HCSYICLO2IMEGVISUXUEQGKNAHV52VDAGDCAP3RKQ) context deadline exceeded; piecestore: (Node ID: 1if7vf1cNTBwAhb6owi6tuGLYqSfa3feWhsvGSevwuwykhz2RY, Piece ID: CFMB4CDU7DWOF3L5T7DASMJB33RVAYSTMM4WFNUSUNV4H6CGPNXQ) context deadline exceeded; piecestore: (Node ID: 1ArQdwHzHSm7c2QZkTt1kQsfzocvCNG2D9yoQgnmbS7HYLCXhy, Piece ID: IY7G4GQVUEYHNO4HJAD3CHGX4SQQNGWPMUXUJJKLC7MUTUUFGFWA) context deadline exceeded ======LOGS====== 4909 , piece_id=EPHUILAEI77HCSYICLO2IMEGVISUXUEQGKNAHV52VDAGDCAP3RKQ, seg_pos={0 0}, upload=39772]: Uploading piece... Dec 5 14:09:11.669180895 (45h13m48.140391634s) [noise=true , address=77.90.31.12:4616 , piece_id=SUX7UBSIQIIYU327MGSXKLJBPXFW6UEVI3NJRAWVEZ3GBUN2QCJQ, seg_pos={0 0}, upload=39772]: Uploading piece... Dec 5 14:09:11.669641381 (45h13m48.14085212s) [noise=true , address=145.79.227.32:4855 , piece_id=N3F2LVAV3UNBWZ7V7HFLNVU5NGWD6X2RB6WKVEVGBJMHUUIJP3PA, seg_pos={0 0}, upload=39772]: Uploading piece... Dec 5 14:09:11.670111447 (45h13m48.141322196s) [noise=true , address=77.90.31.17:4794 , piece_id=ALZXJQTN64GGUOZPJMGJ6CRXQBO2R5PRVNUBQHRKWYGY7KJF6CVQ, seg_pos={0 0}, upload=39772]: Uploading piece... Dec 5 14:09:11.670601259 (45h13m48.141811999s) [noise=true , address=64.49.13.169:4705 , piece_id=TTCHF4ENHBWLOJQCAJKMG6BUZGHDDEU3KW5AU5Q3CFCOTKIONE4Q, seg_pos={0 0}, upload=39772]: Uploading piece... Dec 5 14:09:11.671075502 (45h13m48.142286241s) [noise=true , address=77.90.31.13:4654 , piece_id=CEAQAXPEBEPYGOMBFNVRRYRCT67SMMNEVDBADWIOPQXEJCN3EGZQ, seg_pos={0 0}, upload=39772]: Uploading piece... Dec 5 14:09:11.671598037 (45h13m48.142808776s) [noise=true , address=45.91.116.11:4147 , piece_id=UALY4LTNPIS62L46OIH4PTKRES72I7LA55UL7B2UCZLRT7DQ7RVQ, seg_pos={0 0}, upload=39772]: Uploading piece... Dec 5 14:09:11.672127346 (45h13m48.143338105s) [noise=true , address=145.79.227.15:4907 , piece_id=BXEPFHK6AENJZM2APP5LPV2LHJ4L7CBFQF5JSQJQBC6F5AKDGYQA, seg_pos={0 0}, upload=39772]: Uploading piece... Dec 5 14:09:11.672648588 (45h13m48.143859327s) [noise=true , address=77.90.31.18:4838 , piece_id=CFMB4CDU7DWOF3L5T7DASMJB33RVAYSTMM4WFNUSUNV4H6CGPNXQ, seg_pos={0 0}, upload=39772]: Uploading piece... Dec 5 14:09:11.673155192 (45h13m48.144365931s) [noise=true , address=77.90.31.19:4856 , piece_id=MQGKTFSYGQSOMNUEIVOGGS5ZEVM52CLW4TEG7DA5ICOK73GEAI5Q, seg_pos={0 0}, upload=39772]: Uploading piece... Dec 5 14:09:11.673691684 (45h13m48.144902423s) [noise=true , address=45.91.116.10:4091 , piece_id=ZS3JXULFI4AGS3HIIX2POYQ3XLBG6GPFAWBAV2NBRLMO447DH7ZA, seg_pos={0 0}, upload=39772]: Uploading piece... Dec 5 14:09:11.674805966 (45h13m48.146016705s) [noise=true , address=145.79.227.19:4097 , piece_id=MKAKS25RPIEVHT2WJJ7QYY4FBS7JCIM3RKY7MWRJ7FZTZ2YZXMGQ, seg_pos={0 0}, upload=39772]: Uploading piece... Dec 5 14:09:11.675360772 (45h13m48.146571511s) [noise=true , address=45.91.116.5:4329 , piece_id=MBHG2UWXQMEZPJJHW67KJMJTIVT4QWPXKDIRBRVTBAFN46J3M42Q, seg_pos={0 0}, upload=39772]: Uploading piece... Dec 5 14:09:13.239845356 (45h13m49.711056085s) [upload=39772]: Next returned nil segment Dec 5 14:09:13.239860604 (45h13m49.711071334s) [upload=39772]: Waiting for error group managing segments... Dec 5 14:09:13.643777330 (45h13m50.114988069s) [noise=true , address=77.90.31.13:4654 , piece_id=CEAQAXPEBEPYGOMBFNVRRYRCT67SMMNEVDBADWIOPQXEJCN3EGZQ, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil> Dec 5 14:09:13.697999800 (45h13m50.169210539s) [noise=true , address=77.90.31.18:4838 , piece_id=CFMB4CDU7DWOF3L5T7DASMJB33RVAYSTMM4WFNUSUNV4H6CGPNXQ, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil> Dec 5 14:09:13.713127946 (45h13m50.184338685s) [noise=true , address=77.90.31.15:4729 , piece_id=IY7G4GQVUEYHNO4HJAD3CHGX4SQQNGWPMUXUJJKLC7MUTUUFGFWA, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil> Dec 5 14:09:13.718498752 (45h13m50.189709491s) [noise=true , address=77.90.31.21:4909 , piece_id=EPHUILAEI77HCSYICLO2IMEGVISUXUEQGKNAHV52VDAGDCAP3RKQ, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil> Dec 5 14:09:13.726522451 (45h13m50.19773318s) [noise=true , address=77.90.31.17:4794 , piece_id=ALZXJQTN64GGUOZPJMGJ6CRXQBO2R5PRVNUBQHRKWYGY7KJF6CVQ, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil> Dec 5 14:09:13.749057442 (45h13m50.220268171s) [noise=true , address=45.91.116.10:4091 , piece_id=ZS3JXULFI4AGS3HIIX2POYQ3XLBG6GPFAWBAV2NBRLMO447DH7ZA, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil> Dec 5 14:09:13.750808776 (45h13m50.222019515s) [noise=true , address=77.90.31.10:4542 , piece_id=XDD4QZAKCWJK5A3F5FDL4NRNPEQZF3GKODCGVT2V47AH6KVEOOWQ, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil> Dec 5 14:09:13.754762756 (45h13m50.225973485s) [noise=true , address=77.90.31.11:4556 , piece_id=WBCLEOB7XBAT2MPT6M7NLLNQM2JR3S22BV6UZBSGRDFIJP5JS2LA, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil> Dec 5 14:09:13.759242185 (45h13m50.230452925s) [noise=true , address=77.90.31.12:4616 , piece_id=SUX7UBSIQIIYU327MGSXKLJBPXFW6UEVI3NJRAWVEZ3GBUN2QCJQ, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil> Dec 5 14:09:13.774028982 (45h13m50.245239711s) [noise=true , address=77.90.31.19:4856 , piece_id=MQGKTFSYGQSOMNUEIVOGGS5ZEVM52CLW4TEG7DA5ICOK73GEAI5Q, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil> Dec 5 14:09:13.786684680 (45h13m50.257895419s) [noise=true , address=45.91.116.9:4272 , piece_id=4W72IA43JPPLQ5ZDZHZDW6AFEZDVG7AHJT7555P4BY2Z3QYMLKNA, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil> Dec 5 14:09:13.790669208 (45h13m50.261879947s) [noise=true , address=45.91.116.11:4147 , piece_id=UALY4LTNPIS62L46OIH4PTKRES72I7LA55UL7B2UCZLRT7DQ7RVQ, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil> Dec 5 14:09:13.850747877 (45h13m50.321958617s) [noise=true , address=45.91.116.8:4071 , piece_id=3YI67CEXFW75VRROYLSABR55DNVFHE4SR6EUP57CPRIIB63GZOVA, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil> Dec 5 14:09:13.884817071 (45h13m50.3560278s) [noise=true , address=45.91.116.5:4329 , piece_id=MBHG2UWXQMEZPJJHW67KJMJTIVT4QWPXKDIRBRVTBAFN46J3M42Q, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil> Dec 5 14:09:13.884838361 (45h13m50.35604909s) [seg_pos={0 0}, upload=39772]: Segment reached optimal threshold of 14 pieces. Dec 5 14:09:13.885054584 (45h13m50.356265313s) [noise=true , address=145.79.227.15:4907 , piece_id=BXEPFHK6AENJZM2APP5LPV2LHJ4L7CBFQF5JSQJQBC6F5AKDGYQA, seg_pos={0 0}, upload=39772]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JNee734A635XWMM3Q3N6siDsvKY3FEMQerDjYZrQR6orGBaY3): protocol: write tcp 188.166.225.170:59612->145.79.227.15:4907: use of closed network connection; write tcp 188.166.225.170:59612->145.79.227.15:4907: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59612->145.79.227.15:4907: use of closed network connection Dec 5 14:09:13.885064302 (45h13m50.356275041s) [noise=true , address=145.79.227.15:4907 , piece_id=BXEPFHK6AENJZM2APP5LPV2LHJ4L7CBFQF5JSQJQBC6F5AKDGYQA, seg_pos={0 0}, upload=39772]: Long tail cancellation due to optimal threshold Dec 5 14:09:13.885416693 (45h13m50.356627432s) [noise=true , address=145.79.227.19:4097 , piece_id=MKAKS25RPIEVHT2WJJ7QYY4FBS7JCIM3RKY7MWRJ7FZTZ2YZXMGQ, seg_pos={0 0}, upload=39772]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QP2aZ9BC22cD1Nzz2E12DiA1XoNu3GGTgHC8Cc3TvpQM3Ubn3): protocol: write tcp 188.166.225.170:37344->145.79.227.19:4097: use of closed network connection; write tcp 188.166.225.170:37344->145.79.227.19:4097: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37344->145.79.227.19:4097: use of closed network connection Dec 5 14:09:13.885427744 (45h13m50.356638483s) [noise=true , address=145.79.227.19:4097 , piece_id=MKAKS25RPIEVHT2WJJ7QYY4FBS7JCIM3RKY7MWRJ7FZTZ2YZXMGQ, seg_pos={0 0}, upload=39772]: Long tail cancellation due to optimal threshold Dec 5 14:09:13.885527824 (45h13m50.356738563s) [noise=true , address=145.79.227.32:4855 , piece_id=N3F2LVAV3UNBWZ7V7HFLNVU5NGWD6X2RB6WKVEVGBJMHUUIJP3PA, seg_pos={0 0}, upload=39772]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TGTRhEi2KsbrkC6ZmgGkMaJri9uhzEcmXe1J4nJjpEd1BE4ka): protocol: write tcp 188.166.225.170:43992->145.79.227.32:4855: use of closed network connection; write tcp 188.166.225.170:43992->145.79.227.32:4855: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43992->145.79.227.32:4855: use of closed network connection Dec 5 14:09:13.885535649 (45h13m50.356746388s) [noise=true , address=145.79.227.32:4855 , piece_id=N3F2LVAV3UNBWZ7V7HFLNVU5NGWD6X2RB6WKVEVGBJMHUUIJP3PA, seg_pos={0 0}, upload=39772]: Long tail cancellation due to optimal threshold Dec 5 14:09:13.885614951 (45h13m50.35682569s) [noise=true , address=64.49.13.169:4705 , piece_id=TTCHF4ENHBWLOJQCAJKMG6BUZGHDDEU3KW5AU5Q3CFCOTKIONE4Q, seg_pos={0 0}, upload=39772]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:44254->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:44254->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44254->64.49.13.169:4705: use of closed network connection Dec 5 14:09:13.885620010 (45h13m50.356830749s) [noise=true , address=64.49.13.169:4705 , piece_id=TTCHF4ENHBWLOJQCAJKMG6BUZGHDDEU3KW5AU5Q3CFCOTKIONE4Q, seg_pos={0 0}, upload=39772]: Long tail cancellation due to optimal threshold Dec 5 14:09:13.885636241 (45h13m50.35684697s) [seg_pos={0 0}, upload=39772]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 5 14:09:13.885731973 (45h13m50.356942722s) [upload=39772]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 5 14:09:13.885739618 (45h13m50.356950357s) [upload=39772]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 5 14:09:14.639752295 (45h13m51.110963024s) [upload=39772]: Done uploading segments... Dec 5 14:09:14.639788193 (45h13m51.110998932s) [upload=39772]: Upload finished. err: <nil> Duration: 8.961708721s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 2.962758255s trace: https://jaeger.dev.storj.io/trace/41b9575307efc1e3 FAIL (2025-12-05 14:29:11 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 2.962789145s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 6.946696788s trace: https://jaeger.dev.storj.io/trace/572c91a59c970da7 FAIL (2025-12-05 14:29:04 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 6.946723107s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 2.894262648s trace: https://jaeger.dev.storj.io/trace/361a2ea17a62744e FAIL (2025-12-05 14:29:01 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 5 14:08:02.649750072 (45h12m39.120960811s) [upload=39762]: Starting upload Dec 5 14:08:02.650012922 (45h12m39.121223681s) [upload=39762]: Done starting upload Dec 5 14:08:02.650140385 (45h12m39.121351124s) [upload=39762]: Uploading segments... Dec 5 14:08:02.650204036 (45h12m39.121414775s) [upload=39762]: Got next segment. Inline: true Dec 5 14:08:02.650210659 (45h12m39.121421398s) [upload=39762]: Waiting for error group managing segments... Dec 5 14:08:02.650285611 (45h12m39.12149635s) [upload=39762]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 5 14:08:02.650292053 (45h12m39.121502792s) [upload=39762]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Dec 5 14:08:02.650295811 (45h12m39.12150655s) [upload=39762]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 5 14:08:05.223576463 (45h12m41.694787212s) [upload=39762]: Done uploading segments... Dec 5 14:08:05.223604205 (45h12m41.694814934s) [upload=39762]: Upload finished. err: <nil> Duration:* 2.894291895s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 9.179758787s trace: https://jaeger.dev.storj.io/trace/65df260343fa2062 FAIL (2025-12-05 14:28:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, Piece ID: 2KY6XA6532G23SRYSNBYNYZ2Y65ZLMXDC2ERGQ6PH4UJQZDUZRUA) context deadline exceeded; piecestore: (Node ID: 12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, Piece ID: JYAT6OFRJQNEZPORFRHUCTFQDPVM2ZTXEUBWGNY6BLADCTZIS66Q) context deadline exceeded; piecestore: (Node ID: 1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, Piece ID: RV4CB7VT7T3YVFFBGXA6EVYMGEOWBU3C3XYUTEMRZDBVHRNVEVLA) context deadline exceeded; piecestore: (Node ID: 12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, Piece ID: VMP5DCNJDH5EAO6LOHSOCP4GGOYI6EV3YHCW5GXB2BRTOO67YQMQ) context deadline exceeded; piecestore: (Node ID: 1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5, Piece ID: UNWONHZ6EIRFROZ5L5YAEFBSMFZHEN4VKAKYM2DWTW4BGCTNMAMA) context deadline exceeded ======LOGS====== Dec 5 13:50:13.914958542 (44h54m50.386169281s) [upload=39497]: Starting upload Dec 5 13:50:13.915042042 (44h54m50.386252771s) [upload=39497]: Done starting upload Dec 5 13:50:13.915237493 (44h54m50.386448232s) [upload=39497]: Uploading segments... Dec 5 13:50:13.915339918 (44h54m50.386550688s) [upload=39497]: Got next segment. Inline: false Dec 5 13:50:13.915395575 (44h54m50.386606314s) [upload=39497]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 5 13:50:13.915400735 (44h54m50.386611464s) [upload=39497]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 5 13:50:14.177967782 (44h54m50.649178531s) [seg_pos={0 0}, upload=39497]: Begin upload segment... Dec 5 13:50:14.178049498 (44h54m50.649260237s) [seg_pos={0 0}, upload=39497]: Done begin upload segment. Dec 5 13:50:14.178089053 (44h54m50.649299792s) [noise=true , address=145.79.227.34:4906 , piece_id=L2SWU22TLNZNS3LO2E3HYCKOVPO6CIOZWOACPQV7CGMURN2HEQRQ, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.178100966 (44h54m50.649311715s) [noise=true , address=145.79.227.10:4402 , piece_id=IHNZYV6LJREQ63GBFJY6BVB54QLSVCZIIUUTLOG7YOOOXNGHQ7WA, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.178155129 (44h54m50.649365868s) [noise=true , address=145.79.227.23:4546 , piece_id=UNWONHZ6EIRFROZ5L5YAEFBSMFZHEN4VKAKYM2DWTW4BGCTNMAMA, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.178737678 (44h54m50.649948417s) [noise=true , address=145.79.227.39:4015 , piece_id=V6WH4XCY2YVP2B3DR4HLI7IFH7MGGPAIBAZHHDBHQQ4NUTCNHP7Q, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.179277977 (44h54m50.650488716s) [noise=true , address=145.79.227.21:4474 , piece_id=J5V4QU72P7D2FGZ7NWO5UT3E7S25CTN4MCSPQMSOWK7R6GK4B4HA, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.179826452 (44h54m50.651037191s) [noise=true , address=145.79.227.21:4475 , piece_id=5IPJJ5RHSXCJPUUXXD62EKMNSWSHPIVEWWYS4X77GE7OT6NCTJUA, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.180516445 (44h54m50.651727174s) [noise=true , address=145.79.227.38:4978 , piece_id=N7OVLKMMMZW75CCZDX5QYOG2SDCO7AKO2FITHFDO3EMDSPJJPAYA, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.181184327 (44h54m50.652395066s) [noise=true , address=145.79.227.43:4157 , piece_id=PFPPMUR22PUIHIPCNXQGN5CYFM4K4OG2EMC5LAMUH26RZKETHZAQ, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.182617817 (44h54m50.653828556s) [noise=true , address=145.79.227.41:4087 , piece_id=2KY6XA6532G23SRYSNBYNYZ2Y65ZLMXDC2ERGQ6PH4UJQZDUZRUA, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.183276120 (44h54m50.654486859s) [noise=true , address=145.79.227.16:4909 , piece_id=JYAT6OFRJQNEZPORFRHUCTFQDPVM2ZTXEUBWGNY6BLADCTZIS66Q, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.184534556 (44h54m50.655745295s) [noise=true , address=145.79.227.20:4438 , piece_id=P64J3PKA4V3SQWAC36KWBVXDBE4KXYJHUXWYHNFZHCQEGQZUWHFQ, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.186480883 (44h54m50.657691622s) [noise=true , address=145.79.227.25:4619 , piece_id=RV4CB7VT7T3YVFFBGXA6EVYMGEOWBU3C3XYUTEMRZDBVHRNVEVLA, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.188448450 (44h54m50.659659189s) [noise=true , address=77.90.31.14:4663 , piece_id=7WW2DVMZSNJTG3J4NANUAIZKMLICAYLP5RAOKKWCTCUTJAISG7SQ, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.190696029 (44h54m50.661906768s) [noise=true , address=145.79.227.4:4187 , piece_id=VMP5DCNJDH5EAO6LOHSOCP4GGOYI6EV3YHCW5GXB2BRTOO67YQMQ, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.193175661 (44h54m50.6643864s) [noise=true , address=145.79.227.31:4799 , piece_id=2C63VGDP7JVQVWSLYLHZEHC4SB7PU6NFUZWLCTEO7AGDP32X3JOQ, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.195649231 (44h54m50.66685997s) [noise=true , address=145.79.227.14:4838 , piece_id=6EVD7RSXQJOFT645D7ILWLUWUBYFP6IRR5GESYOAWXF2IXKJEUCA, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:15.979218426 (44h54m52.450429165s) [upload=39497]: Next returned nil segment Dec 5 13:50:15.979231121 (44h54m52.45044185s) [upload=39497]: Waiting for error group managing segments... Dec 5 13:50:16.323526524 (44h54m52.794737263s) [noise=true , address=145.79.227.21:4475 , piece_id=5IPJJ5RHSXCJPUUXXD62EKMNSWSHPIVEWWYS4X77GE7OT6NCTJUA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:16.347685978 (44h54m52.818896717s) [noise=true , address=145.79.227.23:4546 , piece_id=UNWONHZ6EIRFROZ5L5YAEFBSMFZHEN4VKAKYM2DWTW4BGCTNMAMA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:16.351351307 (44h54m52.822562036s) [noise=true , address=145.79.227.10:4402 , piece_id=IHNZYV6LJREQ63GBFJY6BVB54QLSVCZIIUUTLOG7YOOOXNGHQ7WA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:16.351375874 (44h54m52.822586603s) [noise=true , address=145.79.227.38:4978 , piece_id=N7OVLKMMMZW75CCZDX5QYOG2SDCO7AKO2FITHFDO3EMDSPJJPAYA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:16.359501778 (44h54m52.830712517s) [noise=true , address=145.79.227.31:4799 , piece_id=2C63VGDP7JVQVWSLYLHZEHC4SB7PU6NFUZWLCTEO7AGDP32X3JOQ, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:16.360729425 (44h54m52.831940164s) [noise=true , address=145.79.227.4:4187 , piece_id=VMP5DCNJDH5EAO6LOHSOCP4GGOYI6EV3YHCW5GXB2BRTOO67YQMQ, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:16.371054476 (44h54m52.842265225s) [noise=true , address=145.79.227.41:4087 , piece_id=2KY6XA6532G23SRYSNBYNYZ2Y65ZLMXDC2ERGQ6PH4UJQZDUZRUA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:16.392350999 (44h54m52.863561738s) [noise=true , address=77.90.31.14:4663 , piece_id=7WW2DVMZSNJTG3J4NANUAIZKMLICAYLP5RAOKKWCTCUTJAISG7SQ, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:16.437887268 (44h54m52.909098007s) [noise=true , address=145.79.227.21:4474 , piece_id=J5V4QU72P7D2FGZ7NWO5UT3E7S25CTN4MCSPQMSOWK7R6GK4B4HA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:16.642787347 (44h54m53.113998086s) [noise=true , address=145.79.227.43:4157 , piece_id=PFPPMUR22PUIHIPCNXQGN5CYFM4K4OG2EMC5LAMUH26RZKETHZAQ, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:16.987804677 (44h54m53.459015416s) [noise=true , address=145.79.227.25:4619 , piece_id=RV4CB7VT7T3YVFFBGXA6EVYMGEOWBU3C3XYUTEMRZDBVHRNVEVLA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:17.014774723 (44h54m53.485985462s) [noise=true , address=145.79.227.14:4838 , piece_id=6EVD7RSXQJOFT645D7ILWLUWUBYFP6IRR5GESYOAWXF2IXKJEUCA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:17.413923466 (44h54m53.885134205s) [noise=true , address=145.79.227.16:4909 , piece_id=JYAT6OFRJQNEZPORFRHUCTFQDPVM2ZTXEUBWGNY6BLADCTZIS66Q, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:17.413981627 (44h54m53.885192367s) [seg_pos={0 0}, upload=39497]: Segment reached optimal threshold of 13 pieces. Dec 5 13:50:17.414271349 (44h54m53.885482088s) [noise=true , address=145.79.227.39:4015 , piece_id=V6WH4XCY2YVP2B3DR4HLI7IFH7MGGPAIBAZHHDBHQQ4NUTCNHP7Q, seg_pos={0 0}, upload=39497]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE): protocol: write tcp 188.166.225.170:56230->145.79.227.39:4015: use of closed network connection; write tcp 188.166.225.170:56230->145.79.227.39:4015: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56230->145.79.227.39:4015: use of closed network connection Dec 5 13:50:17.414282390 (44h54m53.885493119s) [noise=true , address=145.79.227.39:4015 , piece_id=V6WH4XCY2YVP2B3DR4HLI7IFH7MGGPAIBAZHHDBHQQ4NUTCNHP7Q, seg_pos={0 0}, upload=39497]: Long tail cancellation due to optimal threshold Dec 5 13:50:17.414355008 (44h54m53.885565747s) [noise=true , address=145.79.227.34:4906 , piece_id=L2SWU22TLNZNS3LO2E3HYCKOVPO6CIOZWOACPQV7CGMURN2HEQRQ, seg_pos={0 0}, upload=39497]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 5 13:50:17.414365318 (44h54m53.885576057s) [noise=true , address=145.79.227.34:4906 , piece_id=L2SWU22TLNZNS3LO2E3HYCKOVPO6CIOZWOACPQV7CGMURN2HEQRQ, seg_pos={0 0}, upload=39497]: Long tail cancellation due to optimal threshold Dec 5 13:50:17.414811999 (44h54m53.886022758s) [noise=true , address=145.79.227.20:4438 , piece_id=P64J3PKA4V3SQWAC36KWBVXDBE4KXYJHUXWYHNFZHCQEGQZUWHFQ, seg_pos={0 0}, upload=39497]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT): protocol: expected piece hash; context canceled; EOF Dec 5 13:50:17.414824472 (44h54m53.886035211s) [noise=true , address=145.79.227.20:4438 , piece_id=P64J3PKA4V3SQWAC36KWBVXDBE4KXYJHUXWYHNFZHCQEGQZUWHFQ, seg_pos={0 0}, upload=39497]: Long tail cancellation due to optimal threshold Dec 5 13:50:17.414850933 (44h54m53.886061672s) [seg_pos={0 0}, upload=39497]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 5 13:50:17.414965771 (44h54m53.88617652s) [upload=39497]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 5 13:50:17.414973846 (44h54m53.886184585s) [upload=39497]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 5 13:50:17.720870703 (44h54m54.192081452s) [upload=39497]: Done uploading segments... Dec 5 13:50:17.720894969 (44h54m54.192105708s) [upload=39497]: Upload finished. err: <nil> Duration: 9.179782732s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 2.792983233s trace: https://jaeger.dev.storj.io/trace/41b9575307efc1e3 FAIL (2025-12-05 14:28:43 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 2.793013983s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 47 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXX.X..XXXXX..XXXXXX.X.XXXXXXX.XXX.XXXXX.XXX.X..] duration: 5.001296928s trace: https://jaeger.dev.storj.io/trace/75d5f40d5cfca95c FAIL (2025-12-05 14:28:43 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001322678s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 2.783294135s trace: https://jaeger.dev.storj.io/trace/361a2ea17a62744e FAIL (2025-12-05 14:28:37 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 5 14:08:02.649750072 (45h12m39.120960811s) [upload=39762]: Starting upload Dec 5 14:08:02.650012922 (45h12m39.121223681s) [upload=39762]: Done starting upload Dec 5 14:08:02.650140385 (45h12m39.121351124s) [upload=39762]: Uploading segments... Dec 5 14:08:02.650204036 (45h12m39.121414775s) [upload=39762]: Got next segment. Inline: true Dec 5 14:08:02.650210659 (45h12m39.121421398s) [upload=39762]: Waiting for error group managing segments... Dec 5 14:08:02.650285611 (45h12m39.12149635s) [upload=39762]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 5 14:08:02.650292053 (45h12m39.121502792s) [upload=39762]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Dec 5 14:08:02.650295811 (45h12m39.12150655s) [upload=39762]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 5 14:08:05.223576463 (45h12m41.694787212s) [upload=39762]: Done uploading segments... Dec 5 14:08:05.223604205 (45h12m41.694814934s) [upload=39762]: Upload finished. err: <nil> Duration:* 2.783324272s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [X.......X....................................X..............] duration: 19.825067015s trace: https://jaeger.dev.storj.io/trace/496d6a85c2db53eb FAIL (2025-12-05 14:28:24 UTC) orbiter warning: context deadline exceeded Duration: 19.825117901s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 2.860362056s trace: https://jaeger.dev.storj.io/trace/41b9575307efc1e3 FAIL (2025-12-05 14:28:21 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 2.860398725s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 8.278552967s trace: https://jaeger.dev.storj.io/trace/70e54e58d52fa95e FAIL (2025-12-05 14:28:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1MntjJtcCDSRozKpesp6S2ZjYrMhxiQheDiPMPSCnavWBBGYt3, Piece ID: MQGKTFSYGQSOMNUEIVOGGS5ZEVM52CLW4TEG7DA5ICOK73GEAI5Q) context deadline exceeded; piecestore: (Node ID: 12YnWetw1NereHQi7cwHCnmu1LDjRFNpB2MQHSWQrjQdN1CZbMw, Piece ID: UALY4LTNPIS62L46OIH4PTKRES72I7LA55UL7B2UCZLRT7DQ7RVQ) context deadline exceeded; piecestore: (Node ID: 12Dfm6iHQbMhj1YWLu3UboTCtZ2hq87aiYUgH7Kig3pDUrPeFj4, Piece ID: 4W72IA43JPPLQ5ZDZHZDW6AFEZDVG7AHJT7555P4BY2Z3QYMLKNA) context deadline exceeded; piecestore: (Node ID: 12Qb6sMhCyzYd4S15Td1kzD7kgqNKSi2avG15CQRSG8bcDVvGzi, Piece ID: SUX7UBSIQIIYU327MGSXKLJBPXFW6UEVI3NJRAWVEZ3GBUN2QCJQ) context deadline exceeded; piecestore: (Node ID: 1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks, Piece ID: WBCLEOB7XBAT2MPT6M7NLLNQM2JR3S22BV6UZBSGRDFIJP5JS2LA) context deadline exceeded; piecestore: (Node ID: 123QEF1LQm8nDCTjZvq51BjSMTYwHjhj2ajZd7bCRjiWJVLGXyS, Piece ID: MBHG2UWXQMEZPJJHW67KJMJTIVT4QWPXKDIRBRVTBAFN46J3M42Q) context deadline exceeded; piecestore: (Node ID: 1gPa2A8V3hY6WFgDTMk5ZMQpEaVVuhjXFQYHoZhK2R9JUPyepz, Piece ID: ZS3JXULFI4AGS3HIIX2POYQ3XLBG6GPFAWBAV2NBRLMO447DH7ZA) context deadline exceeded; piecestore: (Node ID: 121RT7tP6czMFAiZ3DjzSSsYgPt3xZXVbXVpkcYxaZYscNiMjL6, Piece ID: EPHUILAEI77HCSYICLO2IMEGVISUXUEQGKNAHV52VDAGDCAP3RKQ) context deadline exceeded; piecestore: (Node ID: 1if7vf1cNTBwAhb6owi6tuGLYqSfa3feWhsvGSevwuwykhz2RY, Piece ID: CFMB4CDU7DWOF3L5T7DASMJB33RVAYSTMM4WFNUSUNV4H6CGPNXQ) context deadline exceeded; piecestore: (Node ID: 1ArQdwHzHSm7c2QZkTt1kQsfzocvCNG2D9yoQgnmbS7HYLCXhy, Piece ID: IY7G4GQVUEYHNO4HJAD3CHGX4SQQNGWPMUXUJJKLC7MUTUUFGFWA) context deadline exceeded ======LOGS====== 4909 , piece_id=EPHUILAEI77HCSYICLO2IMEGVISUXUEQGKNAHV52VDAGDCAP3RKQ, seg_pos={0 0}, upload=39772]: Uploading piece... Dec 5 14:09:11.669180895 (45h13m48.140391634s) [noise=true , address=77.90.31.12:4616 , piece_id=SUX7UBSIQIIYU327MGSXKLJBPXFW6UEVI3NJRAWVEZ3GBUN2QCJQ, seg_pos={0 0}, upload=39772]: Uploading piece... Dec 5 14:09:11.669641381 (45h13m48.14085212s) [noise=true , address=145.79.227.32:4855 , piece_id=N3F2LVAV3UNBWZ7V7HFLNVU5NGWD6X2RB6WKVEVGBJMHUUIJP3PA, seg_pos={0 0}, upload=39772]: Uploading piece... Dec 5 14:09:11.670111447 (45h13m48.141322196s) [noise=true , address=77.90.31.17:4794 , piece_id=ALZXJQTN64GGUOZPJMGJ6CRXQBO2R5PRVNUBQHRKWYGY7KJF6CVQ, seg_pos={0 0}, upload=39772]: Uploading piece... Dec 5 14:09:11.670601259 (45h13m48.141811999s) [noise=true , address=64.49.13.169:4705 , piece_id=TTCHF4ENHBWLOJQCAJKMG6BUZGHDDEU3KW5AU5Q3CFCOTKIONE4Q, seg_pos={0 0}, upload=39772]: Uploading piece... Dec 5 14:09:11.671075502 (45h13m48.142286241s) [noise=true , address=77.90.31.13:4654 , piece_id=CEAQAXPEBEPYGOMBFNVRRYRCT67SMMNEVDBADWIOPQXEJCN3EGZQ, seg_pos={0 0}, upload=39772]: Uploading piece... Dec 5 14:09:11.671598037 (45h13m48.142808776s) [noise=true , address=45.91.116.11:4147 , piece_id=UALY4LTNPIS62L46OIH4PTKRES72I7LA55UL7B2UCZLRT7DQ7RVQ, seg_pos={0 0}, upload=39772]: Uploading piece... Dec 5 14:09:11.672127346 (45h13m48.143338105s) [noise=true , address=145.79.227.15:4907 , piece_id=BXEPFHK6AENJZM2APP5LPV2LHJ4L7CBFQF5JSQJQBC6F5AKDGYQA, seg_pos={0 0}, upload=39772]: Uploading piece... Dec 5 14:09:11.672648588 (45h13m48.143859327s) [noise=true , address=77.90.31.18:4838 , piece_id=CFMB4CDU7DWOF3L5T7DASMJB33RVAYSTMM4WFNUSUNV4H6CGPNXQ, seg_pos={0 0}, upload=39772]: Uploading piece... Dec 5 14:09:11.673155192 (45h13m48.144365931s) [noise=true , address=77.90.31.19:4856 , piece_id=MQGKTFSYGQSOMNUEIVOGGS5ZEVM52CLW4TEG7DA5ICOK73GEAI5Q, seg_pos={0 0}, upload=39772]: Uploading piece... Dec 5 14:09:11.673691684 (45h13m48.144902423s) [noise=true , address=45.91.116.10:4091 , piece_id=ZS3JXULFI4AGS3HIIX2POYQ3XLBG6GPFAWBAV2NBRLMO447DH7ZA, seg_pos={0 0}, upload=39772]: Uploading piece... Dec 5 14:09:11.674805966 (45h13m48.146016705s) [noise=true , address=145.79.227.19:4097 , piece_id=MKAKS25RPIEVHT2WJJ7QYY4FBS7JCIM3RKY7MWRJ7FZTZ2YZXMGQ, seg_pos={0 0}, upload=39772]: Uploading piece... Dec 5 14:09:11.675360772 (45h13m48.146571511s) [noise=true , address=45.91.116.5:4329 , piece_id=MBHG2UWXQMEZPJJHW67KJMJTIVT4QWPXKDIRBRVTBAFN46J3M42Q, seg_pos={0 0}, upload=39772]: Uploading piece... Dec 5 14:09:13.239845356 (45h13m49.711056085s) [upload=39772]: Next returned nil segment Dec 5 14:09:13.239860604 (45h13m49.711071334s) [upload=39772]: Waiting for error group managing segments... Dec 5 14:09:13.643777330 (45h13m50.114988069s) [noise=true , address=77.90.31.13:4654 , piece_id=CEAQAXPEBEPYGOMBFNVRRYRCT67SMMNEVDBADWIOPQXEJCN3EGZQ, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil> Dec 5 14:09:13.697999800 (45h13m50.169210539s) [noise=true , address=77.90.31.18:4838 , piece_id=CFMB4CDU7DWOF3L5T7DASMJB33RVAYSTMM4WFNUSUNV4H6CGPNXQ, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil> Dec 5 14:09:13.713127946 (45h13m50.184338685s) [noise=true , address=77.90.31.15:4729 , piece_id=IY7G4GQVUEYHNO4HJAD3CHGX4SQQNGWPMUXUJJKLC7MUTUUFGFWA, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil> Dec 5 14:09:13.718498752 (45h13m50.189709491s) [noise=true , address=77.90.31.21:4909 , piece_id=EPHUILAEI77HCSYICLO2IMEGVISUXUEQGKNAHV52VDAGDCAP3RKQ, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil> Dec 5 14:09:13.726522451 (45h13m50.19773318s) [noise=true , address=77.90.31.17:4794 , piece_id=ALZXJQTN64GGUOZPJMGJ6CRXQBO2R5PRVNUBQHRKWYGY7KJF6CVQ, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil> Dec 5 14:09:13.749057442 (45h13m50.220268171s) [noise=true , address=45.91.116.10:4091 , piece_id=ZS3JXULFI4AGS3HIIX2POYQ3XLBG6GPFAWBAV2NBRLMO447DH7ZA, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil> Dec 5 14:09:13.750808776 (45h13m50.222019515s) [noise=true , address=77.90.31.10:4542 , piece_id=XDD4QZAKCWJK5A3F5FDL4NRNPEQZF3GKODCGVT2V47AH6KVEOOWQ, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil> Dec 5 14:09:13.754762756 (45h13m50.225973485s) [noise=true , address=77.90.31.11:4556 , piece_id=WBCLEOB7XBAT2MPT6M7NLLNQM2JR3S22BV6UZBSGRDFIJP5JS2LA, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil> Dec 5 14:09:13.759242185 (45h13m50.230452925s) [noise=true , address=77.90.31.12:4616 , piece_id=SUX7UBSIQIIYU327MGSXKLJBPXFW6UEVI3NJRAWVEZ3GBUN2QCJQ, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil> Dec 5 14:09:13.774028982 (45h13m50.245239711s) [noise=true , address=77.90.31.19:4856 , piece_id=MQGKTFSYGQSOMNUEIVOGGS5ZEVM52CLW4TEG7DA5ICOK73GEAI5Q, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil> Dec 5 14:09:13.786684680 (45h13m50.257895419s) [noise=true , address=45.91.116.9:4272 , piece_id=4W72IA43JPPLQ5ZDZHZDW6AFEZDVG7AHJT7555P4BY2Z3QYMLKNA, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil> Dec 5 14:09:13.790669208 (45h13m50.261879947s) [noise=true , address=45.91.116.11:4147 , piece_id=UALY4LTNPIS62L46OIH4PTKRES72I7LA55UL7B2UCZLRT7DQ7RVQ, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil> Dec 5 14:09:13.850747877 (45h13m50.321958617s) [noise=true , address=45.91.116.8:4071 , piece_id=3YI67CEXFW75VRROYLSABR55DNVFHE4SR6EUP57CPRIIB63GZOVA, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil> Dec 5 14:09:13.884817071 (45h13m50.3560278s) [noise=true , address=45.91.116.5:4329 , piece_id=MBHG2UWXQMEZPJJHW67KJMJTIVT4QWPXKDIRBRVTBAFN46J3M42Q, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil> Dec 5 14:09:13.884838361 (45h13m50.35604909s) [seg_pos={0 0}, upload=39772]: Segment reached optimal threshold of 14 pieces. Dec 5 14:09:13.885054584 (45h13m50.356265313s) [noise=true , address=145.79.227.15:4907 , piece_id=BXEPFHK6AENJZM2APP5LPV2LHJ4L7CBFQF5JSQJQBC6F5AKDGYQA, seg_pos={0 0}, upload=39772]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JNee734A635XWMM3Q3N6siDsvKY3FEMQerDjYZrQR6orGBaY3): protocol: write tcp 188.166.225.170:59612->145.79.227.15:4907: use of closed network connection; write tcp 188.166.225.170:59612->145.79.227.15:4907: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59612->145.79.227.15:4907: use of closed network connection Dec 5 14:09:13.885064302 (45h13m50.356275041s) [noise=true , address=145.79.227.15:4907 , piece_id=BXEPFHK6AENJZM2APP5LPV2LHJ4L7CBFQF5JSQJQBC6F5AKDGYQA, seg_pos={0 0}, upload=39772]: Long tail cancellation due to optimal threshold Dec 5 14:09:13.885416693 (45h13m50.356627432s) [noise=true , address=145.79.227.19:4097 , piece_id=MKAKS25RPIEVHT2WJJ7QYY4FBS7JCIM3RKY7MWRJ7FZTZ2YZXMGQ, seg_pos={0 0}, upload=39772]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QP2aZ9BC22cD1Nzz2E12DiA1XoNu3GGTgHC8Cc3TvpQM3Ubn3): protocol: write tcp 188.166.225.170:37344->145.79.227.19:4097: use of closed network connection; write tcp 188.166.225.170:37344->145.79.227.19:4097: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37344->145.79.227.19:4097: use of closed network connection Dec 5 14:09:13.885427744 (45h13m50.356638483s) [noise=true , address=145.79.227.19:4097 , piece_id=MKAKS25RPIEVHT2WJJ7QYY4FBS7JCIM3RKY7MWRJ7FZTZ2YZXMGQ, seg_pos={0 0}, upload=39772]: Long tail cancellation due to optimal threshold Dec 5 14:09:13.885527824 (45h13m50.356738563s) [noise=true , address=145.79.227.32:4855 , piece_id=N3F2LVAV3UNBWZ7V7HFLNVU5NGWD6X2RB6WKVEVGBJMHUUIJP3PA, seg_pos={0 0}, upload=39772]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TGTRhEi2KsbrkC6ZmgGkMaJri9uhzEcmXe1J4nJjpEd1BE4ka): protocol: write tcp 188.166.225.170:43992->145.79.227.32:4855: use of closed network connection; write tcp 188.166.225.170:43992->145.79.227.32:4855: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43992->145.79.227.32:4855: use of closed network connection Dec 5 14:09:13.885535649 (45h13m50.356746388s) [noise=true , address=145.79.227.32:4855 , piece_id=N3F2LVAV3UNBWZ7V7HFLNVU5NGWD6X2RB6WKVEVGBJMHUUIJP3PA, seg_pos={0 0}, upload=39772]: Long tail cancellation due to optimal threshold Dec 5 14:09:13.885614951 (45h13m50.35682569s) [noise=true , address=64.49.13.169:4705 , piece_id=TTCHF4ENHBWLOJQCAJKMG6BUZGHDDEU3KW5AU5Q3CFCOTKIONE4Q, seg_pos={0 0}, upload=39772]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:44254->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:44254->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44254->64.49.13.169:4705: use of closed network connection Dec 5 14:09:13.885620010 (45h13m50.356830749s) [noise=true , address=64.49.13.169:4705 , piece_id=TTCHF4ENHBWLOJQCAJKMG6BUZGHDDEU3KW5AU5Q3CFCOTKIONE4Q, seg_pos={0 0}, upload=39772]: Long tail cancellation due to optimal threshold Dec 5 14:09:13.885636241 (45h13m50.35684697s) [seg_pos={0 0}, upload=39772]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 5 14:09:13.885731973 (45h13m50.356942722s) [upload=39772]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 5 14:09:13.885739618 (45h13m50.356950357s) [upload=39772]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 5 14:09:14.639752295 (45h13m51.110963024s) [upload=39772]: Done uploading segments... Dec 5 14:09:14.639788193 (45h13m51.110998932s) [upload=39772]: Upload finished. err: <nil> Duration: 8.278607971s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 9.005720029s trace: https://jaeger.dev.storj.io/trace/65df260343fa2062 FAIL (2025-12-05 14:28:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, Piece ID: 2KY6XA6532G23SRYSNBYNYZ2Y65ZLMXDC2ERGQ6PH4UJQZDUZRUA) context deadline exceeded; piecestore: (Node ID: 12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, Piece ID: JYAT6OFRJQNEZPORFRHUCTFQDPVM2ZTXEUBWGNY6BLADCTZIS66Q) context deadline exceeded; piecestore: (Node ID: 1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, Piece ID: RV4CB7VT7T3YVFFBGXA6EVYMGEOWBU3C3XYUTEMRZDBVHRNVEVLA) context deadline exceeded; piecestore: (Node ID: 12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, Piece ID: VMP5DCNJDH5EAO6LOHSOCP4GGOYI6EV3YHCW5GXB2BRTOO67YQMQ) context deadline exceeded; piecestore: (Node ID: 1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5, Piece ID: UNWONHZ6EIRFROZ5L5YAEFBSMFZHEN4VKAKYM2DWTW4BGCTNMAMA) context deadline exceeded ======LOGS====== Dec 5 13:50:13.914958542 (44h54m50.386169281s) [upload=39497]: Starting upload Dec 5 13:50:13.915042042 (44h54m50.386252771s) [upload=39497]: Done starting upload Dec 5 13:50:13.915237493 (44h54m50.386448232s) [upload=39497]: Uploading segments... Dec 5 13:50:13.915339918 (44h54m50.386550688s) [upload=39497]: Got next segment. Inline: false Dec 5 13:50:13.915395575 (44h54m50.386606314s) [upload=39497]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 5 13:50:13.915400735 (44h54m50.386611464s) [upload=39497]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 5 13:50:14.177967782 (44h54m50.649178531s) [seg_pos={0 0}, upload=39497]: Begin upload segment... Dec 5 13:50:14.178049498 (44h54m50.649260237s) [seg_pos={0 0}, upload=39497]: Done begin upload segment. Dec 5 13:50:14.178089053 (44h54m50.649299792s) [noise=true , address=145.79.227.34:4906 , piece_id=L2SWU22TLNZNS3LO2E3HYCKOVPO6CIOZWOACPQV7CGMURN2HEQRQ, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.178100966 (44h54m50.649311715s) [noise=true , address=145.79.227.10:4402 , piece_id=IHNZYV6LJREQ63GBFJY6BVB54QLSVCZIIUUTLOG7YOOOXNGHQ7WA, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.178155129 (44h54m50.649365868s) [noise=true , address=145.79.227.23:4546 , piece_id=UNWONHZ6EIRFROZ5L5YAEFBSMFZHEN4VKAKYM2DWTW4BGCTNMAMA, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.178737678 (44h54m50.649948417s) [noise=true , address=145.79.227.39:4015 , piece_id=V6WH4XCY2YVP2B3DR4HLI7IFH7MGGPAIBAZHHDBHQQ4NUTCNHP7Q, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.179277977 (44h54m50.650488716s) [noise=true , address=145.79.227.21:4474 , piece_id=J5V4QU72P7D2FGZ7NWO5UT3E7S25CTN4MCSPQMSOWK7R6GK4B4HA, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.179826452 (44h54m50.651037191s) [noise=true , address=145.79.227.21:4475 , piece_id=5IPJJ5RHSXCJPUUXXD62EKMNSWSHPIVEWWYS4X77GE7OT6NCTJUA, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.180516445 (44h54m50.651727174s) [noise=true , address=145.79.227.38:4978 , piece_id=N7OVLKMMMZW75CCZDX5QYOG2SDCO7AKO2FITHFDO3EMDSPJJPAYA, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.181184327 (44h54m50.652395066s) [noise=true , address=145.79.227.43:4157 , piece_id=PFPPMUR22PUIHIPCNXQGN5CYFM4K4OG2EMC5LAMUH26RZKETHZAQ, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.182617817 (44h54m50.653828556s) [noise=true , address=145.79.227.41:4087 , piece_id=2KY6XA6532G23SRYSNBYNYZ2Y65ZLMXDC2ERGQ6PH4UJQZDUZRUA, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.183276120 (44h54m50.654486859s) [noise=true , address=145.79.227.16:4909 , piece_id=JYAT6OFRJQNEZPORFRHUCTFQDPVM2ZTXEUBWGNY6BLADCTZIS66Q, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.184534556 (44h54m50.655745295s) [noise=true , address=145.79.227.20:4438 , piece_id=P64J3PKA4V3SQWAC36KWBVXDBE4KXYJHUXWYHNFZHCQEGQZUWHFQ, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.186480883 (44h54m50.657691622s) [noise=true , address=145.79.227.25:4619 , piece_id=RV4CB7VT7T3YVFFBGXA6EVYMGEOWBU3C3XYUTEMRZDBVHRNVEVLA, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.188448450 (44h54m50.659659189s) [noise=true , address=77.90.31.14:4663 , piece_id=7WW2DVMZSNJTG3J4NANUAIZKMLICAYLP5RAOKKWCTCUTJAISG7SQ, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.190696029 (44h54m50.661906768s) [noise=true , address=145.79.227.4:4187 , piece_id=VMP5DCNJDH5EAO6LOHSOCP4GGOYI6EV3YHCW5GXB2BRTOO67YQMQ, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.193175661 (44h54m50.6643864s) [noise=true , address=145.79.227.31:4799 , piece_id=2C63VGDP7JVQVWSLYLHZEHC4SB7PU6NFUZWLCTEO7AGDP32X3JOQ, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.195649231 (44h54m50.66685997s) [noise=true , address=145.79.227.14:4838 , piece_id=6EVD7RSXQJOFT645D7ILWLUWUBYFP6IRR5GESYOAWXF2IXKJEUCA, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:15.979218426 (44h54m52.450429165s) [upload=39497]: Next returned nil segment Dec 5 13:50:15.979231121 (44h54m52.45044185s) [upload=39497]: Waiting for error group managing segments... Dec 5 13:50:16.323526524 (44h54m52.794737263s) [noise=true , address=145.79.227.21:4475 , piece_id=5IPJJ5RHSXCJPUUXXD62EKMNSWSHPIVEWWYS4X77GE7OT6NCTJUA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:16.347685978 (44h54m52.818896717s) [noise=true , address=145.79.227.23:4546 , piece_id=UNWONHZ6EIRFROZ5L5YAEFBSMFZHEN4VKAKYM2DWTW4BGCTNMAMA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:16.351351307 (44h54m52.822562036s) [noise=true , address=145.79.227.10:4402 , piece_id=IHNZYV6LJREQ63GBFJY6BVB54QLSVCZIIUUTLOG7YOOOXNGHQ7WA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:16.351375874 (44h54m52.822586603s) [noise=true , address=145.79.227.38:4978 , piece_id=N7OVLKMMMZW75CCZDX5QYOG2SDCO7AKO2FITHFDO3EMDSPJJPAYA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:16.359501778 (44h54m52.830712517s) [noise=true , address=145.79.227.31:4799 , piece_id=2C63VGDP7JVQVWSLYLHZEHC4SB7PU6NFUZWLCTEO7AGDP32X3JOQ, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:16.360729425 (44h54m52.831940164s) [noise=true , address=145.79.227.4:4187 , piece_id=VMP5DCNJDH5EAO6LOHSOCP4GGOYI6EV3YHCW5GXB2BRTOO67YQMQ, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:16.371054476 (44h54m52.842265225s) [noise=true , address=145.79.227.41:4087 , piece_id=2KY6XA6532G23SRYSNBYNYZ2Y65ZLMXDC2ERGQ6PH4UJQZDUZRUA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:16.392350999 (44h54m52.863561738s) [noise=true , address=77.90.31.14:4663 , piece_id=7WW2DVMZSNJTG3J4NANUAIZKMLICAYLP5RAOKKWCTCUTJAISG7SQ, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:16.437887268 (44h54m52.909098007s) [noise=true , address=145.79.227.21:4474 , piece_id=J5V4QU72P7D2FGZ7NWO5UT3E7S25CTN4MCSPQMSOWK7R6GK4B4HA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:16.642787347 (44h54m53.113998086s) [noise=true , address=145.79.227.43:4157 , piece_id=PFPPMUR22PUIHIPCNXQGN5CYFM4K4OG2EMC5LAMUH26RZKETHZAQ, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:16.987804677 (44h54m53.459015416s) [noise=true , address=145.79.227.25:4619 , piece_id=RV4CB7VT7T3YVFFBGXA6EVYMGEOWBU3C3XYUTEMRZDBVHRNVEVLA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:17.014774723 (44h54m53.485985462s) [noise=true , address=145.79.227.14:4838 , piece_id=6EVD7RSXQJOFT645D7ILWLUWUBYFP6IRR5GESYOAWXF2IXKJEUCA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:17.413923466 (44h54m53.885134205s) [noise=true , address=145.79.227.16:4909 , piece_id=JYAT6OFRJQNEZPORFRHUCTFQDPVM2ZTXEUBWGNY6BLADCTZIS66Q, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:17.413981627 (44h54m53.885192367s) [seg_pos={0 0}, upload=39497]: Segment reached optimal threshold of 13 pieces. Dec 5 13:50:17.414271349 (44h54m53.885482088s) [noise=true , address=145.79.227.39:4015 , piece_id=V6WH4XCY2YVP2B3DR4HLI7IFH7MGGPAIBAZHHDBHQQ4NUTCNHP7Q, seg_pos={0 0}, upload=39497]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE): protocol: write tcp 188.166.225.170:56230->145.79.227.39:4015: use of closed network connection; write tcp 188.166.225.170:56230->145.79.227.39:4015: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56230->145.79.227.39:4015: use of closed network connection Dec 5 13:50:17.414282390 (44h54m53.885493119s) [noise=true , address=145.79.227.39:4015 , piece_id=V6WH4XCY2YVP2B3DR4HLI7IFH7MGGPAIBAZHHDBHQQ4NUTCNHP7Q, seg_pos={0 0}, upload=39497]: Long tail cancellation due to optimal threshold Dec 5 13:50:17.414355008 (44h54m53.885565747s) [noise=true , address=145.79.227.34:4906 , piece_id=L2SWU22TLNZNS3LO2E3HYCKOVPO6CIOZWOACPQV7CGMURN2HEQRQ, seg_pos={0 0}, upload=39497]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 5 13:50:17.414365318 (44h54m53.885576057s) [noise=true , address=145.79.227.34:4906 , piece_id=L2SWU22TLNZNS3LO2E3HYCKOVPO6CIOZWOACPQV7CGMURN2HEQRQ, seg_pos={0 0}, upload=39497]: Long tail cancellation due to optimal threshold Dec 5 13:50:17.414811999 (44h54m53.886022758s) [noise=true , address=145.79.227.20:4438 , piece_id=P64J3PKA4V3SQWAC36KWBVXDBE4KXYJHUXWYHNFZHCQEGQZUWHFQ, seg_pos={0 0}, upload=39497]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT): protocol: expected piece hash; context canceled; EOF Dec 5 13:50:17.414824472 (44h54m53.886035211s) [noise=true , address=145.79.227.20:4438 , piece_id=P64J3PKA4V3SQWAC36KWBVXDBE4KXYJHUXWYHNFZHCQEGQZUWHFQ, seg_pos={0 0}, upload=39497]: Long tail cancellation due to optimal threshold Dec 5 13:50:17.414850933 (44h54m53.886061672s) [seg_pos={0 0}, upload=39497]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 5 13:50:17.414965771 (44h54m53.88617652s) [upload=39497]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 5 13:50:17.414973846 (44h54m53.886184585s) [upload=39497]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 5 13:50:17.720870703 (44h54m54.192081452s) [upload=39497]: Done uploading segments... Dec 5 13:50:17.720894969 (44h54m54.192105708s) [upload=39497]: Upload finished. err: <nil> Duration: 9.005745877s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 7.17380213s trace: https://jaeger.dev.storj.io/trace/572c91a59c970da7 FAIL (2025-12-05 14:28:03 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 7.17383385s core uplink us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.X.......X....................................X.............] duration: 13.333724031s trace: https://jaeger.dev.storj.io/trace/496d6a85c2db53eb FAIL (2025-12-05 14:27:44 UTC) orbiter warning: context deadline exceeded Duration: 13.333755802s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 48 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXX.X..XXXXX..XXXXXX.X.XXXXXXX.XXX.XXXXX.XXX.X.] duration: 5.001937157s trace: https://jaeger.dev.storj.io/trace/75d5f40d5cfca95c FAIL (2025-12-05 14:27:38 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001963567s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 2.825851711s trace: https://jaeger.dev.storj.io/trace/361a2ea17a62744e FAIL (2025-12-05 14:27:33 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 5 14:08:02.649750072 (45h12m39.120960811s) [upload=39762]: Starting upload Dec 5 14:08:02.650012922 (45h12m39.121223681s) [upload=39762]: Done starting upload Dec 5 14:08:02.650140385 (45h12m39.121351124s) [upload=39762]: Uploading segments... Dec 5 14:08:02.650204036 (45h12m39.121414775s) [upload=39762]: Got next segment. Inline: true Dec 5 14:08:02.650210659 (45h12m39.121421398s) [upload=39762]: Waiting for error group managing segments... Dec 5 14:08:02.650285611 (45h12m39.12149635s) [upload=39762]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 5 14:08:02.650292053 (45h12m39.121502792s) [upload=39762]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Dec 5 14:08:02.650295811 (45h12m39.12150655s) [upload=39762]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 5 14:08:05.223576463 (45h12m41.694787212s) [upload=39762]: Done uploading segments... Dec 5 14:08:05.223604205 (45h12m41.694814934s) [upload=39762]: Upload finished. err: <nil> Duration:* 2.825880387s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 7.26488239s trace: https://jaeger.dev.storj.io/trace/572c91a59c970da7 FAIL (2025-12-05 14:27:28 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 7.264908549s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 5.914392589s trace: https://jaeger.dev.storj.io/trace/361a2ea17a62744e FAIL (2025-12-05 14:27:12 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 5 14:08:02.649750072 (45h12m39.120960811s) [upload=39762]: Starting upload Dec 5 14:08:02.650012922 (45h12m39.121223681s) [upload=39762]: Done starting upload Dec 5 14:08:02.650140385 (45h12m39.121351124s) [upload=39762]: Uploading segments... Dec 5 14:08:02.650204036 (45h12m39.121414775s) [upload=39762]: Got next segment. Inline: true Dec 5 14:08:02.650210659 (45h12m39.121421398s) [upload=39762]: Waiting for error group managing segments... Dec 5 14:08:02.650285611 (45h12m39.12149635s) [upload=39762]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 5 14:08:02.650292053 (45h12m39.121502792s) [upload=39762]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Dec 5 14:08:02.650295811 (45h12m39.12150655s) [upload=39762]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 5 14:08:05.223576463 (45h12m41.694787212s) [upload=39762]: Done uploading segments... Dec 5 14:08:05.223604205 (45h12m41.694814934s) [upload=39762]: Upload finished. err: <nil> Duration:* 5.914418609s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 2.809099942s trace: https://jaeger.dev.storj.io/trace/41b9575307efc1e3 FAIL (2025-12-05 14:27:11 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 2.809134439s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 18.230707811s trace: https://jaeger.dev.storj.io/trace/65df260343fa2062 FAIL (2025-12-05 14:27:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, Piece ID: 2KY6XA6532G23SRYSNBYNYZ2Y65ZLMXDC2ERGQ6PH4UJQZDUZRUA) context deadline exceeded; piecestore: (Node ID: 12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, Piece ID: JYAT6OFRJQNEZPORFRHUCTFQDPVM2ZTXEUBWGNY6BLADCTZIS66Q) context deadline exceeded; piecestore: (Node ID: 1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, Piece ID: RV4CB7VT7T3YVFFBGXA6EVYMGEOWBU3C3XYUTEMRZDBVHRNVEVLA) context deadline exceeded; piecestore: (Node ID: 12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, Piece ID: VMP5DCNJDH5EAO6LOHSOCP4GGOYI6EV3YHCW5GXB2BRTOO67YQMQ) context deadline exceeded; piecestore: (Node ID: 1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5, Piece ID: UNWONHZ6EIRFROZ5L5YAEFBSMFZHEN4VKAKYM2DWTW4BGCTNMAMA) context deadline exceeded ======LOGS====== Dec 5 13:50:13.914958542 (44h54m50.386169281s) [upload=39497]: Starting upload Dec 5 13:50:13.915042042 (44h54m50.386252771s) [upload=39497]: Done starting upload Dec 5 13:50:13.915237493 (44h54m50.386448232s) [upload=39497]: Uploading segments... Dec 5 13:50:13.915339918 (44h54m50.386550688s) [upload=39497]: Got next segment. Inline: false Dec 5 13:50:13.915395575 (44h54m50.386606314s) [upload=39497]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 5 13:50:13.915400735 (44h54m50.386611464s) [upload=39497]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 5 13:50:14.177967782 (44h54m50.649178531s) [seg_pos={0 0}, upload=39497]: Begin upload segment... Dec 5 13:50:14.178049498 (44h54m50.649260237s) [seg_pos={0 0}, upload=39497]: Done begin upload segment. Dec 5 13:50:14.178089053 (44h54m50.649299792s) [noise=true , address=145.79.227.34:4906 , piece_id=L2SWU22TLNZNS3LO2E3HYCKOVPO6CIOZWOACPQV7CGMURN2HEQRQ, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.178100966 (44h54m50.649311715s) [noise=true , address=145.79.227.10:4402 , piece_id=IHNZYV6LJREQ63GBFJY6BVB54QLSVCZIIUUTLOG7YOOOXNGHQ7WA, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.178155129 (44h54m50.649365868s) [noise=true , address=145.79.227.23:4546 , piece_id=UNWONHZ6EIRFROZ5L5YAEFBSMFZHEN4VKAKYM2DWTW4BGCTNMAMA, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.178737678 (44h54m50.649948417s) [noise=true , address=145.79.227.39:4015 , piece_id=V6WH4XCY2YVP2B3DR4HLI7IFH7MGGPAIBAZHHDBHQQ4NUTCNHP7Q, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.179277977 (44h54m50.650488716s) [noise=true , address=145.79.227.21:4474 , piece_id=J5V4QU72P7D2FGZ7NWO5UT3E7S25CTN4MCSPQMSOWK7R6GK4B4HA, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.179826452 (44h54m50.651037191s) [noise=true , address=145.79.227.21:4475 , piece_id=5IPJJ5RHSXCJPUUXXD62EKMNSWSHPIVEWWYS4X77GE7OT6NCTJUA, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.180516445 (44h54m50.651727174s) [noise=true , address=145.79.227.38:4978 , piece_id=N7OVLKMMMZW75CCZDX5QYOG2SDCO7AKO2FITHFDO3EMDSPJJPAYA, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.181184327 (44h54m50.652395066s) [noise=true , address=145.79.227.43:4157 , piece_id=PFPPMUR22PUIHIPCNXQGN5CYFM4K4OG2EMC5LAMUH26RZKETHZAQ, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.182617817 (44h54m50.653828556s) [noise=true , address=145.79.227.41:4087 , piece_id=2KY6XA6532G23SRYSNBYNYZ2Y65ZLMXDC2ERGQ6PH4UJQZDUZRUA, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.183276120 (44h54m50.654486859s) [noise=true , address=145.79.227.16:4909 , piece_id=JYAT6OFRJQNEZPORFRHUCTFQDPVM2ZTXEUBWGNY6BLADCTZIS66Q, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.184534556 (44h54m50.655745295s) [noise=true , address=145.79.227.20:4438 , piece_id=P64J3PKA4V3SQWAC36KWBVXDBE4KXYJHUXWYHNFZHCQEGQZUWHFQ, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.186480883 (44h54m50.657691622s) [noise=true , address=145.79.227.25:4619 , piece_id=RV4CB7VT7T3YVFFBGXA6EVYMGEOWBU3C3XYUTEMRZDBVHRNVEVLA, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.188448450 (44h54m50.659659189s) [noise=true , address=77.90.31.14:4663 , piece_id=7WW2DVMZSNJTG3J4NANUAIZKMLICAYLP5RAOKKWCTCUTJAISG7SQ, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.190696029 (44h54m50.661906768s) [noise=true , address=145.79.227.4:4187 , piece_id=VMP5DCNJDH5EAO6LOHSOCP4GGOYI6EV3YHCW5GXB2BRTOO67YQMQ, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.193175661 (44h54m50.6643864s) [noise=true , address=145.79.227.31:4799 , piece_id=2C63VGDP7JVQVWSLYLHZEHC4SB7PU6NFUZWLCTEO7AGDP32X3JOQ, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.195649231 (44h54m50.66685997s) [noise=true , address=145.79.227.14:4838 , piece_id=6EVD7RSXQJOFT645D7ILWLUWUBYFP6IRR5GESYOAWXF2IXKJEUCA, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:15.979218426 (44h54m52.450429165s) [upload=39497]: Next returned nil segment Dec 5 13:50:15.979231121 (44h54m52.45044185s) [upload=39497]: Waiting for error group managing segments... Dec 5 13:50:16.323526524 (44h54m52.794737263s) [noise=true , address=145.79.227.21:4475 , piece_id=5IPJJ5RHSXCJPUUXXD62EKMNSWSHPIVEWWYS4X77GE7OT6NCTJUA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:16.347685978 (44h54m52.818896717s) [noise=true , address=145.79.227.23:4546 , piece_id=UNWONHZ6EIRFROZ5L5YAEFBSMFZHEN4VKAKYM2DWTW4BGCTNMAMA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:16.351351307 (44h54m52.822562036s) [noise=true , address=145.79.227.10:4402 , piece_id=IHNZYV6LJREQ63GBFJY6BVB54QLSVCZIIUUTLOG7YOOOXNGHQ7WA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:16.351375874 (44h54m52.822586603s) [noise=true , address=145.79.227.38:4978 , piece_id=N7OVLKMMMZW75CCZDX5QYOG2SDCO7AKO2FITHFDO3EMDSPJJPAYA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:16.359501778 (44h54m52.830712517s) [noise=true , address=145.79.227.31:4799 , piece_id=2C63VGDP7JVQVWSLYLHZEHC4SB7PU6NFUZWLCTEO7AGDP32X3JOQ, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:16.360729425 (44h54m52.831940164s) [noise=true , address=145.79.227.4:4187 , piece_id=VMP5DCNJDH5EAO6LOHSOCP4GGOYI6EV3YHCW5GXB2BRTOO67YQMQ, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:16.371054476 (44h54m52.842265225s) [noise=true , address=145.79.227.41:4087 , piece_id=2KY6XA6532G23SRYSNBYNYZ2Y65ZLMXDC2ERGQ6PH4UJQZDUZRUA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:16.392350999 (44h54m52.863561738s) [noise=true , address=77.90.31.14:4663 , piece_id=7WW2DVMZSNJTG3J4NANUAIZKMLICAYLP5RAOKKWCTCUTJAISG7SQ, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:16.437887268 (44h54m52.909098007s) [noise=true , address=145.79.227.21:4474 , piece_id=J5V4QU72P7D2FGZ7NWO5UT3E7S25CTN4MCSPQMSOWK7R6GK4B4HA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:16.642787347 (44h54m53.113998086s) [noise=true , address=145.79.227.43:4157 , piece_id=PFPPMUR22PUIHIPCNXQGN5CYFM4K4OG2EMC5LAMUH26RZKETHZAQ, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:16.987804677 (44h54m53.459015416s) [noise=true , address=145.79.227.25:4619 , piece_id=RV4CB7VT7T3YVFFBGXA6EVYMGEOWBU3C3XYUTEMRZDBVHRNVEVLA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:17.014774723 (44h54m53.485985462s) [noise=true , address=145.79.227.14:4838 , piece_id=6EVD7RSXQJOFT645D7ILWLUWUBYFP6IRR5GESYOAWXF2IXKJEUCA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:17.413923466 (44h54m53.885134205s) [noise=true , address=145.79.227.16:4909 , piece_id=JYAT6OFRJQNEZPORFRHUCTFQDPVM2ZTXEUBWGNY6BLADCTZIS66Q, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:17.413981627 (44h54m53.885192367s) [seg_pos={0 0}, upload=39497]: Segment reached optimal threshold of 13 pieces. Dec 5 13:50:17.414271349 (44h54m53.885482088s) [noise=true , address=145.79.227.39:4015 , piece_id=V6WH4XCY2YVP2B3DR4HLI7IFH7MGGPAIBAZHHDBHQQ4NUTCNHP7Q, seg_pos={0 0}, upload=39497]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE): protocol: write tcp 188.166.225.170:56230->145.79.227.39:4015: use of closed network connection; write tcp 188.166.225.170:56230->145.79.227.39:4015: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56230->145.79.227.39:4015: use of closed network connection Dec 5 13:50:17.414282390 (44h54m53.885493119s) [noise=true , address=145.79.227.39:4015 , piece_id=V6WH4XCY2YVP2B3DR4HLI7IFH7MGGPAIBAZHHDBHQQ4NUTCNHP7Q, seg_pos={0 0}, upload=39497]: Long tail cancellation due to optimal threshold Dec 5 13:50:17.414355008 (44h54m53.885565747s) [noise=true , address=145.79.227.34:4906 , piece_id=L2SWU22TLNZNS3LO2E3HYCKOVPO6CIOZWOACPQV7CGMURN2HEQRQ, seg_pos={0 0}, upload=39497]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 5 13:50:17.414365318 (44h54m53.885576057s) [noise=true , address=145.79.227.34:4906 , piece_id=L2SWU22TLNZNS3LO2E3HYCKOVPO6CIOZWOACPQV7CGMURN2HEQRQ, seg_pos={0 0}, upload=39497]: Long tail cancellation due to optimal threshold Dec 5 13:50:17.414811999 (44h54m53.886022758s) [noise=true , address=145.79.227.20:4438 , piece_id=P64J3PKA4V3SQWAC36KWBVXDBE4KXYJHUXWYHNFZHCQEGQZUWHFQ, seg_pos={0 0}, upload=39497]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT): protocol: expected piece hash; context canceled; EOF Dec 5 13:50:17.414824472 (44h54m53.886035211s) [noise=true , address=145.79.227.20:4438 , piece_id=P64J3PKA4V3SQWAC36KWBVXDBE4KXYJHUXWYHNFZHCQEGQZUWHFQ, seg_pos={0 0}, upload=39497]: Long tail cancellation due to optimal threshold Dec 5 13:50:17.414850933 (44h54m53.886061672s) [seg_pos={0 0}, upload=39497]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 5 13:50:17.414965771 (44h54m53.88617652s) [upload=39497]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 5 13:50:17.414973846 (44h54m53.886184585s) [upload=39497]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 5 13:50:17.720870703 (44h54m54.192081452s) [upload=39497]: Done uploading segments... Dec 5 13:50:17.720894969 (44h54m54.192105708s) [upload=39497]: Upload finished. err: <nil> Duration: 18.230734773s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 8.151118296s trace: https://jaeger.dev.storj.io/trace/70e54e58d52fa95e FAIL (2025-12-05 14:27:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1MntjJtcCDSRozKpesp6S2ZjYrMhxiQheDiPMPSCnavWBBGYt3, Piece ID: MQGKTFSYGQSOMNUEIVOGGS5ZEVM52CLW4TEG7DA5ICOK73GEAI5Q) context deadline exceeded; piecestore: (Node ID: 12YnWetw1NereHQi7cwHCnmu1LDjRFNpB2MQHSWQrjQdN1CZbMw, Piece ID: UALY4LTNPIS62L46OIH4PTKRES72I7LA55UL7B2UCZLRT7DQ7RVQ) context deadline exceeded; piecestore: (Node ID: 12Dfm6iHQbMhj1YWLu3UboTCtZ2hq87aiYUgH7Kig3pDUrPeFj4, Piece ID: 4W72IA43JPPLQ5ZDZHZDW6AFEZDVG7AHJT7555P4BY2Z3QYMLKNA) context deadline exceeded; piecestore: (Node ID: 12Qb6sMhCyzYd4S15Td1kzD7kgqNKSi2avG15CQRSG8bcDVvGzi, Piece ID: SUX7UBSIQIIYU327MGSXKLJBPXFW6UEVI3NJRAWVEZ3GBUN2QCJQ) context deadline exceeded; piecestore: (Node ID: 1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks, Piece ID: WBCLEOB7XBAT2MPT6M7NLLNQM2JR3S22BV6UZBSGRDFIJP5JS2LA) context deadline exceeded; piecestore: (Node ID: 123QEF1LQm8nDCTjZvq51BjSMTYwHjhj2ajZd7bCRjiWJVLGXyS, Piece ID: MBHG2UWXQMEZPJJHW67KJMJTIVT4QWPXKDIRBRVTBAFN46J3M42Q) context deadline exceeded; piecestore: (Node ID: 1gPa2A8V3hY6WFgDTMk5ZMQpEaVVuhjXFQYHoZhK2R9JUPyepz, Piece ID: ZS3JXULFI4AGS3HIIX2POYQ3XLBG6GPFAWBAV2NBRLMO447DH7ZA) context deadline exceeded; piecestore: (Node ID: 121RT7tP6czMFAiZ3DjzSSsYgPt3xZXVbXVpkcYxaZYscNiMjL6, Piece ID: EPHUILAEI77HCSYICLO2IMEGVISUXUEQGKNAHV52VDAGDCAP3RKQ) context deadline exceeded; piecestore: (Node ID: 1if7vf1cNTBwAhb6owi6tuGLYqSfa3feWhsvGSevwuwykhz2RY, Piece ID: CFMB4CDU7DWOF3L5T7DASMJB33RVAYSTMM4WFNUSUNV4H6CGPNXQ) context deadline exceeded; piecestore: (Node ID: 1ArQdwHzHSm7c2QZkTt1kQsfzocvCNG2D9yoQgnmbS7HYLCXhy, Piece ID: IY7G4GQVUEYHNO4HJAD3CHGX4SQQNGWPMUXUJJKLC7MUTUUFGFWA) context deadline exceeded ======LOGS====== 4909 , piece_id=EPHUILAEI77HCSYICLO2IMEGVISUXUEQGKNAHV52VDAGDCAP3RKQ, seg_pos={0 0}, upload=39772]: Uploading piece... Dec 5 14:09:11.669180895 (45h13m48.140391634s) [noise=true , address=77.90.31.12:4616 , piece_id=SUX7UBSIQIIYU327MGSXKLJBPXFW6UEVI3NJRAWVEZ3GBUN2QCJQ, seg_pos={0 0}, upload=39772]: Uploading piece... Dec 5 14:09:11.669641381 (45h13m48.14085212s) [noise=true , address=145.79.227.32:4855 , piece_id=N3F2LVAV3UNBWZ7V7HFLNVU5NGWD6X2RB6WKVEVGBJMHUUIJP3PA, seg_pos={0 0}, upload=39772]: Uploading piece... Dec 5 14:09:11.670111447 (45h13m48.141322196s) [noise=true , address=77.90.31.17:4794 , piece_id=ALZXJQTN64GGUOZPJMGJ6CRXQBO2R5PRVNUBQHRKWYGY7KJF6CVQ, seg_pos={0 0}, upload=39772]: Uploading piece... Dec 5 14:09:11.670601259 (45h13m48.141811999s) [noise=true , address=64.49.13.169:4705 , piece_id=TTCHF4ENHBWLOJQCAJKMG6BUZGHDDEU3KW5AU5Q3CFCOTKIONE4Q, seg_pos={0 0}, upload=39772]: Uploading piece... Dec 5 14:09:11.671075502 (45h13m48.142286241s) [noise=true , address=77.90.31.13:4654 , piece_id=CEAQAXPEBEPYGOMBFNVRRYRCT67SMMNEVDBADWIOPQXEJCN3EGZQ, seg_pos={0 0}, upload=39772]: Uploading piece... Dec 5 14:09:11.671598037 (45h13m48.142808776s) [noise=true , address=45.91.116.11:4147 , piece_id=UALY4LTNPIS62L46OIH4PTKRES72I7LA55UL7B2UCZLRT7DQ7RVQ, seg_pos={0 0}, upload=39772]: Uploading piece... Dec 5 14:09:11.672127346 (45h13m48.143338105s) [noise=true , address=145.79.227.15:4907 , piece_id=BXEPFHK6AENJZM2APP5LPV2LHJ4L7CBFQF5JSQJQBC6F5AKDGYQA, seg_pos={0 0}, upload=39772]: Uploading piece... Dec 5 14:09:11.672648588 (45h13m48.143859327s) [noise=true , address=77.90.31.18:4838 , piece_id=CFMB4CDU7DWOF3L5T7DASMJB33RVAYSTMM4WFNUSUNV4H6CGPNXQ, seg_pos={0 0}, upload=39772]: Uploading piece... Dec 5 14:09:11.673155192 (45h13m48.144365931s) [noise=true , address=77.90.31.19:4856 , piece_id=MQGKTFSYGQSOMNUEIVOGGS5ZEVM52CLW4TEG7DA5ICOK73GEAI5Q, seg_pos={0 0}, upload=39772]: Uploading piece... Dec 5 14:09:11.673691684 (45h13m48.144902423s) [noise=true , address=45.91.116.10:4091 , piece_id=ZS3JXULFI4AGS3HIIX2POYQ3XLBG6GPFAWBAV2NBRLMO447DH7ZA, seg_pos={0 0}, upload=39772]: Uploading piece... Dec 5 14:09:11.674805966 (45h13m48.146016705s) [noise=true , address=145.79.227.19:4097 , piece_id=MKAKS25RPIEVHT2WJJ7QYY4FBS7JCIM3RKY7MWRJ7FZTZ2YZXMGQ, seg_pos={0 0}, upload=39772]: Uploading piece... Dec 5 14:09:11.675360772 (45h13m48.146571511s) [noise=true , address=45.91.116.5:4329 , piece_id=MBHG2UWXQMEZPJJHW67KJMJTIVT4QWPXKDIRBRVTBAFN46J3M42Q, seg_pos={0 0}, upload=39772]: Uploading piece... Dec 5 14:09:13.239845356 (45h13m49.711056085s) [upload=39772]: Next returned nil segment Dec 5 14:09:13.239860604 (45h13m49.711071334s) [upload=39772]: Waiting for error group managing segments... Dec 5 14:09:13.643777330 (45h13m50.114988069s) [noise=true , address=77.90.31.13:4654 , piece_id=CEAQAXPEBEPYGOMBFNVRRYRCT67SMMNEVDBADWIOPQXEJCN3EGZQ, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil> Dec 5 14:09:13.697999800 (45h13m50.169210539s) [noise=true , address=77.90.31.18:4838 , piece_id=CFMB4CDU7DWOF3L5T7DASMJB33RVAYSTMM4WFNUSUNV4H6CGPNXQ, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil> Dec 5 14:09:13.713127946 (45h13m50.184338685s) [noise=true , address=77.90.31.15:4729 , piece_id=IY7G4GQVUEYHNO4HJAD3CHGX4SQQNGWPMUXUJJKLC7MUTUUFGFWA, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil> Dec 5 14:09:13.718498752 (45h13m50.189709491s) [noise=true , address=77.90.31.21:4909 , piece_id=EPHUILAEI77HCSYICLO2IMEGVISUXUEQGKNAHV52VDAGDCAP3RKQ, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil> Dec 5 14:09:13.726522451 (45h13m50.19773318s) [noise=true , address=77.90.31.17:4794 , piece_id=ALZXJQTN64GGUOZPJMGJ6CRXQBO2R5PRVNUBQHRKWYGY7KJF6CVQ, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil> Dec 5 14:09:13.749057442 (45h13m50.220268171s) [noise=true , address=45.91.116.10:4091 , piece_id=ZS3JXULFI4AGS3HIIX2POYQ3XLBG6GPFAWBAV2NBRLMO447DH7ZA, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil> Dec 5 14:09:13.750808776 (45h13m50.222019515s) [noise=true , address=77.90.31.10:4542 , piece_id=XDD4QZAKCWJK5A3F5FDL4NRNPEQZF3GKODCGVT2V47AH6KVEOOWQ, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil> Dec 5 14:09:13.754762756 (45h13m50.225973485s) [noise=true , address=77.90.31.11:4556 , piece_id=WBCLEOB7XBAT2MPT6M7NLLNQM2JR3S22BV6UZBSGRDFIJP5JS2LA, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil> Dec 5 14:09:13.759242185 (45h13m50.230452925s) [noise=true , address=77.90.31.12:4616 , piece_id=SUX7UBSIQIIYU327MGSXKLJBPXFW6UEVI3NJRAWVEZ3GBUN2QCJQ, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil> Dec 5 14:09:13.774028982 (45h13m50.245239711s) [noise=true , address=77.90.31.19:4856 , piece_id=MQGKTFSYGQSOMNUEIVOGGS5ZEVM52CLW4TEG7DA5ICOK73GEAI5Q, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil> Dec 5 14:09:13.786684680 (45h13m50.257895419s) [noise=true , address=45.91.116.9:4272 , piece_id=4W72IA43JPPLQ5ZDZHZDW6AFEZDVG7AHJT7555P4BY2Z3QYMLKNA, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil> Dec 5 14:09:13.790669208 (45h13m50.261879947s) [noise=true , address=45.91.116.11:4147 , piece_id=UALY4LTNPIS62L46OIH4PTKRES72I7LA55UL7B2UCZLRT7DQ7RVQ, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil> Dec 5 14:09:13.850747877 (45h13m50.321958617s) [noise=true , address=45.91.116.8:4071 , piece_id=3YI67CEXFW75VRROYLSABR55DNVFHE4SR6EUP57CPRIIB63GZOVA, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil> Dec 5 14:09:13.884817071 (45h13m50.3560278s) [noise=true , address=45.91.116.5:4329 , piece_id=MBHG2UWXQMEZPJJHW67KJMJTIVT4QWPXKDIRBRVTBAFN46J3M42Q, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil> Dec 5 14:09:13.884838361 (45h13m50.35604909s) [seg_pos={0 0}, upload=39772]: Segment reached optimal threshold of 14 pieces. Dec 5 14:09:13.885054584 (45h13m50.356265313s) [noise=true , address=145.79.227.15:4907 , piece_id=BXEPFHK6AENJZM2APP5LPV2LHJ4L7CBFQF5JSQJQBC6F5AKDGYQA, seg_pos={0 0}, upload=39772]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JNee734A635XWMM3Q3N6siDsvKY3FEMQerDjYZrQR6orGBaY3): protocol: write tcp 188.166.225.170:59612->145.79.227.15:4907: use of closed network connection; write tcp 188.166.225.170:59612->145.79.227.15:4907: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59612->145.79.227.15:4907: use of closed network connection Dec 5 14:09:13.885064302 (45h13m50.356275041s) [noise=true , address=145.79.227.15:4907 , piece_id=BXEPFHK6AENJZM2APP5LPV2LHJ4L7CBFQF5JSQJQBC6F5AKDGYQA, seg_pos={0 0}, upload=39772]: Long tail cancellation due to optimal threshold Dec 5 14:09:13.885416693 (45h13m50.356627432s) [noise=true , address=145.79.227.19:4097 , piece_id=MKAKS25RPIEVHT2WJJ7QYY4FBS7JCIM3RKY7MWRJ7FZTZ2YZXMGQ, seg_pos={0 0}, upload=39772]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QP2aZ9BC22cD1Nzz2E12DiA1XoNu3GGTgHC8Cc3TvpQM3Ubn3): protocol: write tcp 188.166.225.170:37344->145.79.227.19:4097: use of closed network connection; write tcp 188.166.225.170:37344->145.79.227.19:4097: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37344->145.79.227.19:4097: use of closed network connection Dec 5 14:09:13.885427744 (45h13m50.356638483s) [noise=true , address=145.79.227.19:4097 , piece_id=MKAKS25RPIEVHT2WJJ7QYY4FBS7JCIM3RKY7MWRJ7FZTZ2YZXMGQ, seg_pos={0 0}, upload=39772]: Long tail cancellation due to optimal threshold Dec 5 14:09:13.885527824 (45h13m50.356738563s) [noise=true , address=145.79.227.32:4855 , piece_id=N3F2LVAV3UNBWZ7V7HFLNVU5NGWD6X2RB6WKVEVGBJMHUUIJP3PA, seg_pos={0 0}, upload=39772]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TGTRhEi2KsbrkC6ZmgGkMaJri9uhzEcmXe1J4nJjpEd1BE4ka): protocol: write tcp 188.166.225.170:43992->145.79.227.32:4855: use of closed network connection; write tcp 188.166.225.170:43992->145.79.227.32:4855: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43992->145.79.227.32:4855: use of closed network connection Dec 5 14:09:13.885535649 (45h13m50.356746388s) [noise=true , address=145.79.227.32:4855 , piece_id=N3F2LVAV3UNBWZ7V7HFLNVU5NGWD6X2RB6WKVEVGBJMHUUIJP3PA, seg_pos={0 0}, upload=39772]: Long tail cancellation due to optimal threshold Dec 5 14:09:13.885614951 (45h13m50.35682569s) [noise=true , address=64.49.13.169:4705 , piece_id=TTCHF4ENHBWLOJQCAJKMG6BUZGHDDEU3KW5AU5Q3CFCOTKIONE4Q, seg_pos={0 0}, upload=39772]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:44254->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:44254->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44254->64.49.13.169:4705: use of closed network connection Dec 5 14:09:13.885620010 (45h13m50.356830749s) [noise=true , address=64.49.13.169:4705 , piece_id=TTCHF4ENHBWLOJQCAJKMG6BUZGHDDEU3KW5AU5Q3CFCOTKIONE4Q, seg_pos={0 0}, upload=39772]: Long tail cancellation due to optimal threshold Dec 5 14:09:13.885636241 (45h13m50.35684697s) [seg_pos={0 0}, upload=39772]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 5 14:09:13.885731973 (45h13m50.356942722s) [upload=39772]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 5 14:09:13.885739618 (45h13m50.356950357s) [upload=39772]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 5 14:09:14.639752295 (45h13m51.110963024s) [upload=39772]: Done uploading segments... Dec 5 14:09:14.639788193 (45h13m51.110998932s) [upload=39772]: Upload finished. err: <nil> Duration: 8.151139398s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 8.214144853s trace: https://jaeger.dev.storj.io/trace/572c91a59c970da7 FAIL (2025-12-05 14:26:48 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 8.214166885s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 5.070650241s trace: https://jaeger.dev.storj.io/trace/361a2ea17a62744e FAIL (2025-12-05 14:26:42 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 5 14:08:02.649750072 (45h12m39.120960811s) [upload=39762]: Starting upload Dec 5 14:08:02.650012922 (45h12m39.121223681s) [upload=39762]: Done starting upload Dec 5 14:08:02.650140385 (45h12m39.121351124s) [upload=39762]: Uploading segments... Dec 5 14:08:02.650204036 (45h12m39.121414775s) [upload=39762]: Got next segment. Inline: true Dec 5 14:08:02.650210659 (45h12m39.121421398s) [upload=39762]: Waiting for error group managing segments... Dec 5 14:08:02.650285611 (45h12m39.12149635s) [upload=39762]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 5 14:08:02.650292053 (45h12m39.121502792s) [upload=39762]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Dec 5 14:08:02.650295811 (45h12m39.12150655s) [upload=39762]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 5 14:08:05.223576463 (45h12m41.694787212s) [upload=39762]: Done uploading segments... Dec 5 14:08:05.223604205 (45h12m41.694814934s) [upload=39762]: Upload finished. err: <nil> Duration:* 5.070674366s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 2.846916382s trace: https://jaeger.dev.storj.io/trace/41b9575307efc1e3 FAIL (2025-12-05 14:26:38 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 2.84694167s core uplink us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..X.......X....................................X............] duration: 16.028116762s trace: https://jaeger.dev.storj.io/trace/496d6a85c2db53eb FAIL (2025-12-05 14:26:38 UTC) orbiter warning: context deadline exceeded Duration: 16.028156848s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 49 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXX.X..XXXXX..XXXXXX.X.XXXXXXX.XXX.XXXXX.XXX.X] duration: 5.001557826s trace: https://jaeger.dev.storj.io/trace/75d5f40d5cfca95c FAIL (2025-12-05 14:26:35 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00158644s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 2.890288971s trace: https://jaeger.dev.storj.io/trace/41b9575307efc1e3 FAIL (2025-12-05 14:26:21 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 2.890421123s core uplink us1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)