Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 6.353083002s trace: https://jaeger.dev.storj.io/trace/27d92c196f473eec FAIL (2025-12-08 02:35:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 4891s) [upload=51469]: Got next segment. Inline: false Dec 8 02:04:21.143134547 (58h35m21.014217592s) [upload=51469]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 02:04:21.143141259 (58h35m21.014224304s) [upload=51469]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 02:04:22.267502706 (58h35m22.138585751s) [seg_pos={0 0}, upload=51469]: Begin upload segment... Dec 8 02:04:22.267624127 (58h35m22.138707172s) [seg_pos={0 0}, upload=51469]: Done begin upload segment. Dec 8 02:04:22.267698027 (58h35m22.138781073s) [noise=true , address=77.90.31.11:4554 , piece_id=BQP7XTRA2SBVINT4J7JAOF3ORNENBBIQN4ABMCEMQQ6FQL6QUOKA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.267712997 (58h35m22.138796042s) [noise=true , address=145.79.227.41:4087 , piece_id=XO6K3CXK7KZ5VEVXULHKQXLSSLIIWIX3VKQOSD7HY25WMNMCDLEQ, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.268434680 (58h35m22.139517715s) [noise=true , address=145.79.227.16:4910 , piece_id=ETE7ZBHT5IJUOBWCA5BGLKC2ANDEUB2M3DOMXWWS2RBWSJQBARRQ, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.268550331 (58h35m22.139633366s) [noise=true , address=145.79.227.26:4653 , piece_id=DKW72PEWPIOHPYHLI54PBU7NR3HGUBLFM3MN4XQQFWIIZ4FHIUNA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.268913111 (58h35m22.139996146s) [noise=true , address=145.79.227.23:4545 , piece_id=NBYCSU2KSHTAL223AHKHREQWKXUEQKJU46TY3SPYTKT5N5T2OUSA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.268924032 (58h35m22.140007067s) [noise=true , address=145.79.227.40:4051 , piece_id=7CQTGHFZ542WUHYX2CTOPN4PF2KDOOLDGV64THDBZCYOCXA3JWDA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.269398886 (58h35m22.140481921s) [noise=true , address=77.90.31.10:4518 , piece_id=6AR3NAFODP54SKG3SJSJSIQQFOYXO2C65WVNN4Z4XG426ZEOEDQA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.269414686 (58h35m22.140497721s) [noise=true , address=145.79.227.25:4619 , piece_id=XBNGVRIQLQUWKKBF4ULL7ZJDDT75QLY7ES2J7WHYWRD2ZLHOGU2Q, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.269753261 (58h35m22.140836296s) [noise=true , address=145.79.227.23:4547 , piece_id=5Y6V5QWLCWNOLT2GDNUEIRSG75KGNGIFLBCBGSIT7XH46RZYVDGQ, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.270016322 (58h35m22.141099367s) [noise=true , address=77.90.31.13:4628 , piece_id=LWTOCQRRA7HLI5ZPPRQ25IM22LMQNRR5H3TTOTSTGCFOYFZMABAA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.270259175 (58h35m22.14134222s) [noise=true , address=145.79.227.25:4617 , piece_id=S4ETCJ6WMCPXXQCYPPBBMHLMK2SVJV4S5VOKHGTDAGSGG6WRYH3Q, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.270655339 (58h35m22.141738374s) [noise=true , address=145.79.227.38:4977 , piece_id=73WXA33TWM4MFVTHDHW3PXJVABMCVMQYI3P42AHT3OYPKMXIVTUQ, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.271143158 (58h35m22.142246412s) [noise=true , address=145.79.227.28:4727 , piece_id=UYRK457JBF2AHCIXZ4XOIL7YHOVYZ7IPDUTB6WOZZUDEW5Q7NNCQ, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.271230565 (58h35m22.14231361s) [noise=true , address=145.79.227.39:4013 , piece_id=UI22RDXJ2DY3RFNSJVNT43XQJ6B2S3EO6SFLYHXW4VPVQJSYYF6Q, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.271662086 (58h35m22.142745121s) [noise=true , address=145.79.227.22:4511 , piece_id=GGA463DAOSZCW7SQGZW2LGFRQVNUOUJ2L2GXRXLSDPC65KVPY3TA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.272592658 (58h35m22.143675703s) [noise=true , address=145.79.227.24:4583 , piece_id=WYTXGXG77ZW4RGWX7YWDU2NH6R5CCPVBPQH3ISHUAJVKZJRONS5Q, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:24.009199123 (58h35m23.880282158s) [upload=51469]: Next returned nil segment Dec 8 02:04:24.009221235 (58h35m23.88030428s) [upload=51469]: Waiting for error group managing segments... Dec 8 02:04:24.293906896 (58h35m24.164989931s) [noise=true , address=145.79.227.24:4583 , piece_id=WYTXGXG77ZW4RGWX7YWDU2NH6R5CCPVBPQH3ISHUAJVKZJRONS5Q, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.298576738 (58h35m24.169659783s) [noise=true , address=145.79.227.28:4727 , piece_id=UYRK457JBF2AHCIXZ4XOIL7YHOVYZ7IPDUTB6WOZZUDEW5Q7NNCQ, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.301356571 (58h35m24.172439616s) [noise=true , address=145.79.227.25:4619 , piece_id=XBNGVRIQLQUWKKBF4ULL7ZJDDT75QLY7ES2J7WHYWRD2ZLHOGU2Q, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.326466765 (58h35m24.19754981s) [noise=true , address=145.79.227.16:4910 , piece_id=ETE7ZBHT5IJUOBWCA5BGLKC2ANDEUB2M3DOMXWWS2RBWSJQBARRQ, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.328633530 (58h35m24.199716586s) [noise=true , address=145.79.227.26:4653 , piece_id=DKW72PEWPIOHPYHLI54PBU7NR3HGUBLFM3MN4XQQFWIIZ4FHIUNA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.360508106 (58h35m24.231591151s) [noise=true , address=145.79.227.40:4051 , piece_id=7CQTGHFZ542WUHYX2CTOPN4PF2KDOOLDGV64THDBZCYOCXA3JWDA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.361116263 (58h35m24.232199308s) [noise=true , address=145.79.227.25:4617 , piece_id=S4ETCJ6WMCPXXQCYPPBBMHLMK2SVJV4S5VOKHGTDAGSGG6WRYH3Q, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.361695086 (58h35m24.232778131s) [noise=true , address=145.79.227.41:4087 , piece_id=XO6K3CXK7KZ5VEVXULHKQXLSSLIIWIX3VKQOSD7HY25WMNMCDLEQ, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.371520525 (58h35m24.24260356s) [noise=true , address=145.79.227.38:4977 , piece_id=73WXA33TWM4MFVTHDHW3PXJVABMCVMQYI3P42AHT3OYPKMXIVTUQ, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.381271893 (58h35m24.252354928s) [noise=true , address=145.79.227.23:4545 , piece_id=NBYCSU2KSHTAL223AHKHREQWKXUEQKJU46TY3SPYTKT5N5T2OUSA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.386403053 (58h35m24.257486098s) [noise=true , address=77.90.31.11:4554 , piece_id=BQP7XTRA2SBVINT4J7JAOF3ORNENBBIQN4ABMCEMQQ6FQL6QUOKA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.386427209 (58h35m24.257510254s) [noise=true , address=77.90.31.10:4518 , piece_id=6AR3NAFODP54SKG3SJSJSIQQFOYXO2C65WVNN4Z4XG426ZEOEDQA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.400489386 (58h35m24.271572431s) [noise=true , address=77.90.31.13:4628 , piece_id=LWTOCQRRA7HLI5ZPPRQ25IM22LMQNRR5H3TTOTSTGCFOYFZMABAA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.400513903 (58h35m24.271596938s) [seg_pos={0 0}, upload=51469]: Segment reached optimal threshold of 13 pieces. Dec 8 02:04:24.400813233 (58h35m24.271896288s) [noise=true , address=145.79.227.39:4013 , piece_id=UI22RDXJ2DY3RFNSJVNT43XQJ6B2S3EO6SFLYHXW4VPVQJSYYF6Q, seg_pos={0 0}, upload=51469]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b): protocol: write tcp 188.166.225.170:39434->145.79.227.39:4013: use of closed network connection; write tcp 188.166.225.170:39434->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39434->145.79.227.39:4013: use of closed network connection Dec 8 02:04:24.400825677 (58h35m24.271908722s) [noise=true , address=145.79.227.39:4013 , piece_id=UI22RDXJ2DY3RFNSJVNT43XQJ6B2S3EO6SFLYHXW4VPVQJSYYF6Q, seg_pos={0 0}, upload=51469]: Long tail cancellation due to optimal threshold Dec 8 02:04:24.400873027 (58h35m24.271956082s) [noise=true , address=145.79.227.22:4511 , piece_id=GGA463DAOSZCW7SQGZW2LGFRQVNUOUJ2L2GXRXLSDPC65KVPY3TA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG): protocol: write tcp 188.166.225.170:47926->145.79.227.22:4511: use of closed network connection; write tcp 188.166.225.170:47926->145.79.227.22:4511: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47926->145.79.227.22:4511: use of closed network connection Dec 8 02:04:24.400894248 (58h35m24.271977293s) [noise=true , address=145.79.227.22:4511 , piece_id=GGA463DAOSZCW7SQGZW2LGFRQVNUOUJ2L2GXRXLSDPC65KVPY3TA, seg_pos={0 0}, upload=51469]: Long tail cancellation due to optimal threshold Dec 8 02:04:24.401247129 (58h35m24.272330184s) [noise=true , address=145.79.227.23:4547 , piece_id=5Y6V5QWLCWNOLT2GDNUEIRSG75KGNGIFLBCBGSIT7XH46RZYVDGQ, seg_pos={0 0}, upload=51469]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW): protocol: read tcp 188.166.225.170:43344->145.79.227.23:4547: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 02:04:24.401260003 (58h35m24.272343059s) [noise=true , address=145.79.227.23:4547 , piece_id=5Y6V5QWLCWNOLT2GDNUEIRSG75KGNGIFLBCBGSIT7XH46RZYVDGQ, seg_pos={0 0}, upload=51469]: Long tail cancellation due to optimal threshold Dec 8 02:04:24.401285452 (58h35m24.272368497s) [seg_pos={0 0}, upload=51469]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 02:04:24.401382247 (58h35m24.272465292s) [upload=51469]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 02:04:24.401391394 (58h35m24.272474429s) [upload=51469]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 02:05:39.049922629 (58h36m38.921005704s) [upload=51469]: Done uploading segments... Dec 8 02:05:39.050003342 (58h36m38.921086377s) [upload=51469]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.35310777s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 5.724435149s trace: https://jaeger.dev.storj.io/trace/4030003299248e86 FAIL (2025-12-08 02:35:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.724484091s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....X.........X............................................] duration: 5.831169539s trace: https://jaeger.dev.storj.io/trace/e7164007ac0fd55 FAIL (2025-12-08 02:35:17 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.831207059s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [..........X......X................X....X...................X] duration: 5.000587557s trace: https://jaeger.dev.storj.io/trace/527faa4451e27b3d FAIL (2025-12-08 02:35:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000614999s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 4.997102891s trace: https://jaeger.dev.storj.io/trace/7af096edea7ffcf5 FAIL (2025-12-08 02:35:02 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.997135202s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 5.842707518s trace: https://jaeger.dev.storj.io/trace/57258282a8e974fd FAIL (2025-12-08 02:34:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== UA4TAJ2FSYO5S4NXECVMMMCXKFL5CAHPQBQJ6WREQ3YFC3Q, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.202906305 (58h26m7.07398934s) [noise=true , address=45.91.116.4:4945 , piece_id=6UNWQQAWWWWC2XVUCDRCTAYJDTHBXAZJMXA6KLCTOP5PYSAJCCGA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.203371201 (58h26m7.074454246s) [noise=true , address=145.79.227.11:4740 , piece_id=KL54LL2UQRT6MH2K3NQQJB3BKPX4QRWQMXRK27MW2U7CHFDCCWFA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.203861354 (58h26m7.07494439s) [noise=true , address=77.90.31.12:4621 , piece_id=S4FVJHEPIO2MVO56ZRIJ27RMTV723MERPLJLVD2T2GAO3T7K24TA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.204290401 (58h26m7.075373446s) [noise=true , address=145.79.227.34:4928 , piece_id=DRKF2DRSCBAYSZO66A2ZON2JS6HIJIT2YUWFZRS3W2CKFTV3NLVA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.204711363 (58h26m7.075794398s) [noise=true , address=145.79.227.15:4908 , piece_id=JIUZSMOAKBFCFAX6RQ52TCNZMYBREQCYOEM75U3H2QYR4WH3KWCA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.205118208 (58h26m7.076201253s) [noise=true , address=77.90.31.18:4825 , piece_id=EJ4LO4TCEFB4OQQLFOEUYGQKK462ZPXXEIIY5CHF2PERXWDGQNDA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.205554277 (58h26m7.076637323s) [noise=true , address=145.79.227.30:4786 , piece_id=VM65RUN3GUZZFUOHFYE7ZLPWU6COCQUMY3IK7GCNWPIUZAMEBR7Q, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.206116137 (58h26m7.077199182s) [noise=true , address=145.79.227.25:4627 , piece_id=K7UYU3W2CACUAQUUL72GG5F7DP5CHAHZ4M47TZG6NBN5Z6ZCDRBA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.206664562 (58h26m7.077747607s) [noise=true , address=77.90.31.16:4760 , piece_id=Q67LYTDJCAR2TFVSAIIYCUYM7D44JFPBLCSKRNLXH26EAW5UQ3GA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.207102556 (58h26m7.078185601s) [noise=true , address=45.91.116.7:4037 , piece_id=UBNFDSXQBISC5QQSNGAAZIR7LRE7FCYRVYN5MMOY3A2ALEOQBYQQ, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.207519911 (58h26m7.078602956s) [noise=true , address=77.90.31.11:4579 , piece_id=JINRNEVUAP2VI5Y4CR7ALWDQIAIAFX3SP2VVZSXDDG6XPUKASE3Q, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.207967393 (58h26m7.079050438s) [noise=true , address=77.90.31.10:4550 , piece_id=PDCZ7VXNHQJ6LYPG64GZRYP4MRBGVPXCWTXSRT3UV2N6HGMDE2KA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.208416427 (58h26m7.079499472s) [noise=true , address=45.91.116.6:4003 , piece_id=CRONTV2B2AVZTLKU3YNIEGQZSON7XHHL7S3K6KNNJV4YR5FHQ3PA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.208830746 (58h26m7.079913791s) [noise=true , address=45.91.116.12:4254 , piece_id=UCKU4TBXJC4PSYHRDPUXETXLAILDOWA5NN7RCVVNYLVUBY7FBIFA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.209255245 (58h26m7.080338291s) [noise=true , address=145.79.227.16:4933 , piece_id=LHF64L3SWXFVZ5X5D67I44WSJNKRFB5G4YP7DFHDWOAYE23II3OA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.209662921 (58h26m7.080745966s) [noise=true , address=145.79.227.18:4010 , piece_id=S2EOTEV4TBNNDOBN7TIX76AJE2WI5P5DM3JK2NJB25TISYLS7VSQ, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.214493680 (58h26m7.085576725s) [noise=true , address=77.90.31.10:4539 , piece_id=27YFK55QLZDEULYABQ3OBDSPDDEIUJ2MJSD7X3A4K6HRFJMRQJUA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:08.555121654 (58h26m8.426204699s) [upload=51329]: Next returned nil segment Dec 8 01:55:08.555145519 (58h26m8.426228564s) [upload=51329]: Waiting for error group managing segments... Dec 8 01:55:08.935193756 (58h26m8.806276791s) [noise=true , address=64.49.13.27:4957 , piece_id=7METNUA4TAJ2FSYO5S4NXECVMMMCXKFL5CAHPQBQJ6WREQ3YFC3Q, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.093051314 (58h26m8.964134359s) [noise=true , address=145.79.227.16:4933 , piece_id=LHF64L3SWXFVZ5X5D67I44WSJNKRFB5G4YP7DFHDWOAYE23II3OA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.098812273 (58h26m8.969895318s) [noise=true , address=145.79.227.18:4010 , piece_id=S2EOTEV4TBNNDOBN7TIX76AJE2WI5P5DM3JK2NJB25TISYLS7VSQ, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.102616718 (58h26m8.973699763s) [noise=true , address=145.79.227.25:4627 , piece_id=K7UYU3W2CACUAQUUL72GG5F7DP5CHAHZ4M47TZG6NBN5Z6ZCDRBA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.118902949 (58h26m8.989985994s) [noise=true , address=145.79.227.34:4928 , piece_id=DRKF2DRSCBAYSZO66A2ZON2JS6HIJIT2YUWFZRS3W2CKFTV3NLVA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.124783838 (58h26m8.995866883s) [noise=true , address=77.90.31.18:4825 , piece_id=EJ4LO4TCEFB4OQQLFOEUYGQKK462ZPXXEIIY5CHF2PERXWDGQNDA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.126718282 (58h26m8.997801327s) [noise=true , address=145.79.227.11:4740 , piece_id=KL54LL2UQRT6MH2K3NQQJB3BKPX4QRWQMXRK27MW2U7CHFDCCWFA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.145158485 (58h26m9.01624153s) [noise=true , address=77.90.31.10:4550 , piece_id=PDCZ7VXNHQJ6LYPG64GZRYP4MRBGVPXCWTXSRT3UV2N6HGMDE2KA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.156650047 (58h26m9.027733092s) [noise=true , address=145.79.227.15:4908 , piece_id=JIUZSMOAKBFCFAX6RQ52TCNZMYBREQCYOEM75U3H2QYR4WH3KWCA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.162020363 (58h26m9.033103398s) [noise=true , address=77.90.31.10:4539 , piece_id=27YFK55QLZDEULYABQ3OBDSPDDEIUJ2MJSD7X3A4K6HRFJMRQJUA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.198899907 (58h26m9.069982942s) [noise=true , address=77.90.31.12:4621 , piece_id=S4FVJHEPIO2MVO56ZRIJ27RMTV723MERPLJLVD2T2GAO3T7K24TA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.208468337 (58h26m9.079551382s) [noise=true , address=45.91.116.4:4945 , piece_id=6UNWQQAWWWWC2XVUCDRCTAYJDTHBXAZJMXA6KLCTOP5PYSAJCCGA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.212217697 (58h26m9.083300732s) [noise=true , address=45.91.116.12:4254 , piece_id=UCKU4TBXJC4PSYHRDPUXETXLAILDOWA5NN7RCVVNYLVUBY7FBIFA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.215832390 (58h26m9.086915435s) [noise=true , address=77.90.31.11:4579 , piece_id=JINRNEVUAP2VI5Y4CR7ALWDQIAIAFX3SP2VVZSXDDG6XPUKASE3Q, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.215862086 (58h26m9.086945161s) [seg_pos={0 0}, upload=51329]: Segment reached optimal threshold of 14 pieces. Dec 8 01:55:09.216078098 (58h26m9.087161133s) [noise=true , address=45.91.116.6:4003 , piece_id=CRONTV2B2AVZTLKU3YNIEGQZSON7XHHL7S3K6KNNJV4YR5FHQ3PA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1veqoBGw6a8AGMoigyuNJMjcEvCT93xRq2B1d2KNhs6aEmSqVF): protocol: expected piece hash; context canceled; EOF Dec 8 01:55:09.216085662 (58h26m9.087168707s) [noise=true , address=45.91.116.6:4003 , piece_id=CRONTV2B2AVZTLKU3YNIEGQZSON7XHHL7S3K6KNNJV4YR5FHQ3PA, seg_pos={0 0}, upload=51329]: Long tail cancellation due to optimal threshold Dec 8 01:55:09.216307705 (58h26m9.08739075s) [noise=true , address=145.79.227.30:4786 , piece_id=VM65RUN3GUZZFUOHFYE7ZLPWU6COCQUMY3IK7GCNWPIUZAMEBR7Q, seg_pos={0 0}, upload=51329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125TWU7R4pkmrUxsNLhgTD475aLbjUJ7psA6drJtE7oDqZXYJ4Q): protocol: expected piece hash; context canceled; EOF Dec 8 01:55:09.216313956 (58h26m9.087396991s) [noise=true , address=145.79.227.30:4786 , piece_id=VM65RUN3GUZZFUOHFYE7ZLPWU6COCQUMY3IK7GCNWPIUZAMEBR7Q, seg_pos={0 0}, upload=51329]: Long tail cancellation due to optimal threshold Dec 8 01:55:09.216414809 (58h26m9.087497844s) [noise=true , address=77.90.31.16:4760 , piece_id=Q67LYTDJCAR2TFVSAIIYCUYM7D44JFPBLCSKRNLXH26EAW5UQ3GA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1e6p1TD5CqUyWdZjkrxh2kRqMuyDiMUr9eimfeayDAKwhtBrZ8): protocol: expected piece hash; context canceled; EOF Dec 8 01:55:09.216420681 (58h26m9.087503726s) [noise=true , address=77.90.31.16:4760 , piece_id=Q67LYTDJCAR2TFVSAIIYCUYM7D44JFPBLCSKRNLXH26EAW5UQ3GA, seg_pos={0 0}, upload=51329]: Long tail cancellation due to optimal threshold Dec 8 01:55:09.216562861 (58h26m9.087645906s) [noise=true , address=45.91.116.7:4037 , piece_id=UBNFDSXQBISC5QQSNGAAZIR7LRE7FCYRVYN5MMOY3A2ALEOQBYQQ, seg_pos={0 0}, upload=51329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iNPwE2XSfXcaXAXcrH2jcyNGYeDiiWvzAcDQffrmjPwTVqVmK): protocol: expected piece hash; context canceled; EOF Dec 8 01:55:09.216570025 (58h26m9.08765307s) [noise=true , address=45.91.116.7:4037 , piece_id=UBNFDSXQBISC5QQSNGAAZIR7LRE7FCYRVYN5MMOY3A2ALEOQBYQQ, seg_pos={0 0}, upload=51329]: Long tail cancellation due to optimal threshold Dec 8 01:55:09.216585564 (58h26m9.087668609s) [seg_pos={0 0}, upload=51329]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 8 01:55:09.216654365 (58h26m9.0877374s) [upload=51329]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 01:55:09.216659064 (58h26m9.087742099s) [upload=51329]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 01:55:35.852213564 (58h26m35.723296629s) [upload=51329]: Done uploading segments... Dec 8 01:55:35.852273619 (58h26m35.723356664s) [upload=51329]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.842763975s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 5.752809337s trace: https://jaeger.dev.storj.io/trace/49d209695fd399e8 FAIL (2025-12-08 02:34:37 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 5.752833624s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......X.........X...........................................] duration: 6.420049558s trace: https://jaeger.dev.storj.io/trace/e7164007ac0fd55 FAIL (2025-12-08 02:34:35 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 6.420078684s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 5.97066408s trace: https://jaeger.dev.storj.io/trace/27d92c196f473eec FAIL (2025-12-08 02:34:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 4891s) [upload=51469]: Got next segment. Inline: false Dec 8 02:04:21.143134547 (58h35m21.014217592s) [upload=51469]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 02:04:21.143141259 (58h35m21.014224304s) [upload=51469]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 02:04:22.267502706 (58h35m22.138585751s) [seg_pos={0 0}, upload=51469]: Begin upload segment... Dec 8 02:04:22.267624127 (58h35m22.138707172s) [seg_pos={0 0}, upload=51469]: Done begin upload segment. Dec 8 02:04:22.267698027 (58h35m22.138781073s) [noise=true , address=77.90.31.11:4554 , piece_id=BQP7XTRA2SBVINT4J7JAOF3ORNENBBIQN4ABMCEMQQ6FQL6QUOKA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.267712997 (58h35m22.138796042s) [noise=true , address=145.79.227.41:4087 , piece_id=XO6K3CXK7KZ5VEVXULHKQXLSSLIIWIX3VKQOSD7HY25WMNMCDLEQ, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.268434680 (58h35m22.139517715s) [noise=true , address=145.79.227.16:4910 , piece_id=ETE7ZBHT5IJUOBWCA5BGLKC2ANDEUB2M3DOMXWWS2RBWSJQBARRQ, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.268550331 (58h35m22.139633366s) [noise=true , address=145.79.227.26:4653 , piece_id=DKW72PEWPIOHPYHLI54PBU7NR3HGUBLFM3MN4XQQFWIIZ4FHIUNA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.268913111 (58h35m22.139996146s) [noise=true , address=145.79.227.23:4545 , piece_id=NBYCSU2KSHTAL223AHKHREQWKXUEQKJU46TY3SPYTKT5N5T2OUSA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.268924032 (58h35m22.140007067s) [noise=true , address=145.79.227.40:4051 , piece_id=7CQTGHFZ542WUHYX2CTOPN4PF2KDOOLDGV64THDBZCYOCXA3JWDA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.269398886 (58h35m22.140481921s) [noise=true , address=77.90.31.10:4518 , piece_id=6AR3NAFODP54SKG3SJSJSIQQFOYXO2C65WVNN4Z4XG426ZEOEDQA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.269414686 (58h35m22.140497721s) [noise=true , address=145.79.227.25:4619 , piece_id=XBNGVRIQLQUWKKBF4ULL7ZJDDT75QLY7ES2J7WHYWRD2ZLHOGU2Q, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.269753261 (58h35m22.140836296s) [noise=true , address=145.79.227.23:4547 , piece_id=5Y6V5QWLCWNOLT2GDNUEIRSG75KGNGIFLBCBGSIT7XH46RZYVDGQ, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.270016322 (58h35m22.141099367s) [noise=true , address=77.90.31.13:4628 , piece_id=LWTOCQRRA7HLI5ZPPRQ25IM22LMQNRR5H3TTOTSTGCFOYFZMABAA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.270259175 (58h35m22.14134222s) [noise=true , address=145.79.227.25:4617 , piece_id=S4ETCJ6WMCPXXQCYPPBBMHLMK2SVJV4S5VOKHGTDAGSGG6WRYH3Q, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.270655339 (58h35m22.141738374s) [noise=true , address=145.79.227.38:4977 , piece_id=73WXA33TWM4MFVTHDHW3PXJVABMCVMQYI3P42AHT3OYPKMXIVTUQ, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.271143158 (58h35m22.142246412s) [noise=true , address=145.79.227.28:4727 , piece_id=UYRK457JBF2AHCIXZ4XOIL7YHOVYZ7IPDUTB6WOZZUDEW5Q7NNCQ, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.271230565 (58h35m22.14231361s) [noise=true , address=145.79.227.39:4013 , piece_id=UI22RDXJ2DY3RFNSJVNT43XQJ6B2S3EO6SFLYHXW4VPVQJSYYF6Q, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.271662086 (58h35m22.142745121s) [noise=true , address=145.79.227.22:4511 , piece_id=GGA463DAOSZCW7SQGZW2LGFRQVNUOUJ2L2GXRXLSDPC65KVPY3TA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.272592658 (58h35m22.143675703s) [noise=true , address=145.79.227.24:4583 , piece_id=WYTXGXG77ZW4RGWX7YWDU2NH6R5CCPVBPQH3ISHUAJVKZJRONS5Q, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:24.009199123 (58h35m23.880282158s) [upload=51469]: Next returned nil segment Dec 8 02:04:24.009221235 (58h35m23.88030428s) [upload=51469]: Waiting for error group managing segments... Dec 8 02:04:24.293906896 (58h35m24.164989931s) [noise=true , address=145.79.227.24:4583 , piece_id=WYTXGXG77ZW4RGWX7YWDU2NH6R5CCPVBPQH3ISHUAJVKZJRONS5Q, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.298576738 (58h35m24.169659783s) [noise=true , address=145.79.227.28:4727 , piece_id=UYRK457JBF2AHCIXZ4XOIL7YHOVYZ7IPDUTB6WOZZUDEW5Q7NNCQ, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.301356571 (58h35m24.172439616s) [noise=true , address=145.79.227.25:4619 , piece_id=XBNGVRIQLQUWKKBF4ULL7ZJDDT75QLY7ES2J7WHYWRD2ZLHOGU2Q, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.326466765 (58h35m24.19754981s) [noise=true , address=145.79.227.16:4910 , piece_id=ETE7ZBHT5IJUOBWCA5BGLKC2ANDEUB2M3DOMXWWS2RBWSJQBARRQ, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.328633530 (58h35m24.199716586s) [noise=true , address=145.79.227.26:4653 , piece_id=DKW72PEWPIOHPYHLI54PBU7NR3HGUBLFM3MN4XQQFWIIZ4FHIUNA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.360508106 (58h35m24.231591151s) [noise=true , address=145.79.227.40:4051 , piece_id=7CQTGHFZ542WUHYX2CTOPN4PF2KDOOLDGV64THDBZCYOCXA3JWDA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.361116263 (58h35m24.232199308s) [noise=true , address=145.79.227.25:4617 , piece_id=S4ETCJ6WMCPXXQCYPPBBMHLMK2SVJV4S5VOKHGTDAGSGG6WRYH3Q, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.361695086 (58h35m24.232778131s) [noise=true , address=145.79.227.41:4087 , piece_id=XO6K3CXK7KZ5VEVXULHKQXLSSLIIWIX3VKQOSD7HY25WMNMCDLEQ, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.371520525 (58h35m24.24260356s) [noise=true , address=145.79.227.38:4977 , piece_id=73WXA33TWM4MFVTHDHW3PXJVABMCVMQYI3P42AHT3OYPKMXIVTUQ, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.381271893 (58h35m24.252354928s) [noise=true , address=145.79.227.23:4545 , piece_id=NBYCSU2KSHTAL223AHKHREQWKXUEQKJU46TY3SPYTKT5N5T2OUSA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.386403053 (58h35m24.257486098s) [noise=true , address=77.90.31.11:4554 , piece_id=BQP7XTRA2SBVINT4J7JAOF3ORNENBBIQN4ABMCEMQQ6FQL6QUOKA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.386427209 (58h35m24.257510254s) [noise=true , address=77.90.31.10:4518 , piece_id=6AR3NAFODP54SKG3SJSJSIQQFOYXO2C65WVNN4Z4XG426ZEOEDQA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.400489386 (58h35m24.271572431s) [noise=true , address=77.90.31.13:4628 , piece_id=LWTOCQRRA7HLI5ZPPRQ25IM22LMQNRR5H3TTOTSTGCFOYFZMABAA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.400513903 (58h35m24.271596938s) [seg_pos={0 0}, upload=51469]: Segment reached optimal threshold of 13 pieces. Dec 8 02:04:24.400813233 (58h35m24.271896288s) [noise=true , address=145.79.227.39:4013 , piece_id=UI22RDXJ2DY3RFNSJVNT43XQJ6B2S3EO6SFLYHXW4VPVQJSYYF6Q, seg_pos={0 0}, upload=51469]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b): protocol: write tcp 188.166.225.170:39434->145.79.227.39:4013: use of closed network connection; write tcp 188.166.225.170:39434->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39434->145.79.227.39:4013: use of closed network connection Dec 8 02:04:24.400825677 (58h35m24.271908722s) [noise=true , address=145.79.227.39:4013 , piece_id=UI22RDXJ2DY3RFNSJVNT43XQJ6B2S3EO6SFLYHXW4VPVQJSYYF6Q, seg_pos={0 0}, upload=51469]: Long tail cancellation due to optimal threshold Dec 8 02:04:24.400873027 (58h35m24.271956082s) [noise=true , address=145.79.227.22:4511 , piece_id=GGA463DAOSZCW7SQGZW2LGFRQVNUOUJ2L2GXRXLSDPC65KVPY3TA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG): protocol: write tcp 188.166.225.170:47926->145.79.227.22:4511: use of closed network connection; write tcp 188.166.225.170:47926->145.79.227.22:4511: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47926->145.79.227.22:4511: use of closed network connection Dec 8 02:04:24.400894248 (58h35m24.271977293s) [noise=true , address=145.79.227.22:4511 , piece_id=GGA463DAOSZCW7SQGZW2LGFRQVNUOUJ2L2GXRXLSDPC65KVPY3TA, seg_pos={0 0}, upload=51469]: Long tail cancellation due to optimal threshold Dec 8 02:04:24.401247129 (58h35m24.272330184s) [noise=true , address=145.79.227.23:4547 , piece_id=5Y6V5QWLCWNOLT2GDNUEIRSG75KGNGIFLBCBGSIT7XH46RZYVDGQ, seg_pos={0 0}, upload=51469]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW): protocol: read tcp 188.166.225.170:43344->145.79.227.23:4547: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 02:04:24.401260003 (58h35m24.272343059s) [noise=true , address=145.79.227.23:4547 , piece_id=5Y6V5QWLCWNOLT2GDNUEIRSG75KGNGIFLBCBGSIT7XH46RZYVDGQ, seg_pos={0 0}, upload=51469]: Long tail cancellation due to optimal threshold Dec 8 02:04:24.401285452 (58h35m24.272368497s) [seg_pos={0 0}, upload=51469]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 02:04:24.401382247 (58h35m24.272465292s) [upload=51469]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 02:04:24.401391394 (58h35m24.272474429s) [upload=51469]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 02:05:39.049922629 (58h36m38.921005704s) [upload=51469]: Done uploading segments... Dec 8 02:05:39.050003342 (58h36m38.921086377s) [upload=51469]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.970695801s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [...........X......X................X....X...................] duration: 3.384790579s trace: https://jaeger.dev.storj.io/trace/491130a80ad889c4 FAIL (2025-12-08 02:34:22 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.38560543s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 5.898491762s trace: https://jaeger.dev.storj.io/trace/4030003299248e86 FAIL (2025-12-08 02:34:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.898565953s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 4.93442431s trace: https://jaeger.dev.storj.io/trace/7af096edea7ffcf5 FAIL (2025-12-08 02:34:19 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.934455088s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 5.613251645s trace: https://jaeger.dev.storj.io/trace/57258282a8e974fd FAIL (2025-12-08 02:34:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== UA4TAJ2FSYO5S4NXECVMMMCXKFL5CAHPQBQJ6WREQ3YFC3Q, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.202906305 (58h26m7.07398934s) [noise=true , address=45.91.116.4:4945 , piece_id=6UNWQQAWWWWC2XVUCDRCTAYJDTHBXAZJMXA6KLCTOP5PYSAJCCGA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.203371201 (58h26m7.074454246s) [noise=true , address=145.79.227.11:4740 , piece_id=KL54LL2UQRT6MH2K3NQQJB3BKPX4QRWQMXRK27MW2U7CHFDCCWFA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.203861354 (58h26m7.07494439s) [noise=true , address=77.90.31.12:4621 , piece_id=S4FVJHEPIO2MVO56ZRIJ27RMTV723MERPLJLVD2T2GAO3T7K24TA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.204290401 (58h26m7.075373446s) [noise=true , address=145.79.227.34:4928 , piece_id=DRKF2DRSCBAYSZO66A2ZON2JS6HIJIT2YUWFZRS3W2CKFTV3NLVA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.204711363 (58h26m7.075794398s) [noise=true , address=145.79.227.15:4908 , piece_id=JIUZSMOAKBFCFAX6RQ52TCNZMYBREQCYOEM75U3H2QYR4WH3KWCA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.205118208 (58h26m7.076201253s) [noise=true , address=77.90.31.18:4825 , piece_id=EJ4LO4TCEFB4OQQLFOEUYGQKK462ZPXXEIIY5CHF2PERXWDGQNDA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.205554277 (58h26m7.076637323s) [noise=true , address=145.79.227.30:4786 , piece_id=VM65RUN3GUZZFUOHFYE7ZLPWU6COCQUMY3IK7GCNWPIUZAMEBR7Q, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.206116137 (58h26m7.077199182s) [noise=true , address=145.79.227.25:4627 , piece_id=K7UYU3W2CACUAQUUL72GG5F7DP5CHAHZ4M47TZG6NBN5Z6ZCDRBA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.206664562 (58h26m7.077747607s) [noise=true , address=77.90.31.16:4760 , piece_id=Q67LYTDJCAR2TFVSAIIYCUYM7D44JFPBLCSKRNLXH26EAW5UQ3GA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.207102556 (58h26m7.078185601s) [noise=true , address=45.91.116.7:4037 , piece_id=UBNFDSXQBISC5QQSNGAAZIR7LRE7FCYRVYN5MMOY3A2ALEOQBYQQ, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.207519911 (58h26m7.078602956s) [noise=true , address=77.90.31.11:4579 , piece_id=JINRNEVUAP2VI5Y4CR7ALWDQIAIAFX3SP2VVZSXDDG6XPUKASE3Q, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.207967393 (58h26m7.079050438s) [noise=true , address=77.90.31.10:4550 , piece_id=PDCZ7VXNHQJ6LYPG64GZRYP4MRBGVPXCWTXSRT3UV2N6HGMDE2KA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.208416427 (58h26m7.079499472s) [noise=true , address=45.91.116.6:4003 , piece_id=CRONTV2B2AVZTLKU3YNIEGQZSON7XHHL7S3K6KNNJV4YR5FHQ3PA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.208830746 (58h26m7.079913791s) [noise=true , address=45.91.116.12:4254 , piece_id=UCKU4TBXJC4PSYHRDPUXETXLAILDOWA5NN7RCVVNYLVUBY7FBIFA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.209255245 (58h26m7.080338291s) [noise=true , address=145.79.227.16:4933 , piece_id=LHF64L3SWXFVZ5X5D67I44WSJNKRFB5G4YP7DFHDWOAYE23II3OA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.209662921 (58h26m7.080745966s) [noise=true , address=145.79.227.18:4010 , piece_id=S2EOTEV4TBNNDOBN7TIX76AJE2WI5P5DM3JK2NJB25TISYLS7VSQ, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.214493680 (58h26m7.085576725s) [noise=true , address=77.90.31.10:4539 , piece_id=27YFK55QLZDEULYABQ3OBDSPDDEIUJ2MJSD7X3A4K6HRFJMRQJUA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:08.555121654 (58h26m8.426204699s) [upload=51329]: Next returned nil segment Dec 8 01:55:08.555145519 (58h26m8.426228564s) [upload=51329]: Waiting for error group managing segments... Dec 8 01:55:08.935193756 (58h26m8.806276791s) [noise=true , address=64.49.13.27:4957 , piece_id=7METNUA4TAJ2FSYO5S4NXECVMMMCXKFL5CAHPQBQJ6WREQ3YFC3Q, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.093051314 (58h26m8.964134359s) [noise=true , address=145.79.227.16:4933 , piece_id=LHF64L3SWXFVZ5X5D67I44WSJNKRFB5G4YP7DFHDWOAYE23II3OA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.098812273 (58h26m8.969895318s) [noise=true , address=145.79.227.18:4010 , piece_id=S2EOTEV4TBNNDOBN7TIX76AJE2WI5P5DM3JK2NJB25TISYLS7VSQ, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.102616718 (58h26m8.973699763s) [noise=true , address=145.79.227.25:4627 , piece_id=K7UYU3W2CACUAQUUL72GG5F7DP5CHAHZ4M47TZG6NBN5Z6ZCDRBA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.118902949 (58h26m8.989985994s) [noise=true , address=145.79.227.34:4928 , piece_id=DRKF2DRSCBAYSZO66A2ZON2JS6HIJIT2YUWFZRS3W2CKFTV3NLVA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.124783838 (58h26m8.995866883s) [noise=true , address=77.90.31.18:4825 , piece_id=EJ4LO4TCEFB4OQQLFOEUYGQKK462ZPXXEIIY5CHF2PERXWDGQNDA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.126718282 (58h26m8.997801327s) [noise=true , address=145.79.227.11:4740 , piece_id=KL54LL2UQRT6MH2K3NQQJB3BKPX4QRWQMXRK27MW2U7CHFDCCWFA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.145158485 (58h26m9.01624153s) [noise=true , address=77.90.31.10:4550 , piece_id=PDCZ7VXNHQJ6LYPG64GZRYP4MRBGVPXCWTXSRT3UV2N6HGMDE2KA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.156650047 (58h26m9.027733092s) [noise=true , address=145.79.227.15:4908 , piece_id=JIUZSMOAKBFCFAX6RQ52TCNZMYBREQCYOEM75U3H2QYR4WH3KWCA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.162020363 (58h26m9.033103398s) [noise=true , address=77.90.31.10:4539 , piece_id=27YFK55QLZDEULYABQ3OBDSPDDEIUJ2MJSD7X3A4K6HRFJMRQJUA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.198899907 (58h26m9.069982942s) [noise=true , address=77.90.31.12:4621 , piece_id=S4FVJHEPIO2MVO56ZRIJ27RMTV723MERPLJLVD2T2GAO3T7K24TA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.208468337 (58h26m9.079551382s) [noise=true , address=45.91.116.4:4945 , piece_id=6UNWQQAWWWWC2XVUCDRCTAYJDTHBXAZJMXA6KLCTOP5PYSAJCCGA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.212217697 (58h26m9.083300732s) [noise=true , address=45.91.116.12:4254 , piece_id=UCKU4TBXJC4PSYHRDPUXETXLAILDOWA5NN7RCVVNYLVUBY7FBIFA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.215832390 (58h26m9.086915435s) [noise=true , address=77.90.31.11:4579 , piece_id=JINRNEVUAP2VI5Y4CR7ALWDQIAIAFX3SP2VVZSXDDG6XPUKASE3Q, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.215862086 (58h26m9.086945161s) [seg_pos={0 0}, upload=51329]: Segment reached optimal threshold of 14 pieces. Dec 8 01:55:09.216078098 (58h26m9.087161133s) [noise=true , address=45.91.116.6:4003 , piece_id=CRONTV2B2AVZTLKU3YNIEGQZSON7XHHL7S3K6KNNJV4YR5FHQ3PA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1veqoBGw6a8AGMoigyuNJMjcEvCT93xRq2B1d2KNhs6aEmSqVF): protocol: expected piece hash; context canceled; EOF Dec 8 01:55:09.216085662 (58h26m9.087168707s) [noise=true , address=45.91.116.6:4003 , piece_id=CRONTV2B2AVZTLKU3YNIEGQZSON7XHHL7S3K6KNNJV4YR5FHQ3PA, seg_pos={0 0}, upload=51329]: Long tail cancellation due to optimal threshold Dec 8 01:55:09.216307705 (58h26m9.08739075s) [noise=true , address=145.79.227.30:4786 , piece_id=VM65RUN3GUZZFUOHFYE7ZLPWU6COCQUMY3IK7GCNWPIUZAMEBR7Q, seg_pos={0 0}, upload=51329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125TWU7R4pkmrUxsNLhgTD475aLbjUJ7psA6drJtE7oDqZXYJ4Q): protocol: expected piece hash; context canceled; EOF Dec 8 01:55:09.216313956 (58h26m9.087396991s) [noise=true , address=145.79.227.30:4786 , piece_id=VM65RUN3GUZZFUOHFYE7ZLPWU6COCQUMY3IK7GCNWPIUZAMEBR7Q, seg_pos={0 0}, upload=51329]: Long tail cancellation due to optimal threshold Dec 8 01:55:09.216414809 (58h26m9.087497844s) [noise=true , address=77.90.31.16:4760 , piece_id=Q67LYTDJCAR2TFVSAIIYCUYM7D44JFPBLCSKRNLXH26EAW5UQ3GA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1e6p1TD5CqUyWdZjkrxh2kRqMuyDiMUr9eimfeayDAKwhtBrZ8): protocol: expected piece hash; context canceled; EOF Dec 8 01:55:09.216420681 (58h26m9.087503726s) [noise=true , address=77.90.31.16:4760 , piece_id=Q67LYTDJCAR2TFVSAIIYCUYM7D44JFPBLCSKRNLXH26EAW5UQ3GA, seg_pos={0 0}, upload=51329]: Long tail cancellation due to optimal threshold Dec 8 01:55:09.216562861 (58h26m9.087645906s) [noise=true , address=45.91.116.7:4037 , piece_id=UBNFDSXQBISC5QQSNGAAZIR7LRE7FCYRVYN5MMOY3A2ALEOQBYQQ, seg_pos={0 0}, upload=51329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iNPwE2XSfXcaXAXcrH2jcyNGYeDiiWvzAcDQffrmjPwTVqVmK): protocol: expected piece hash; context canceled; EOF Dec 8 01:55:09.216570025 (58h26m9.08765307s) [noise=true , address=45.91.116.7:4037 , piece_id=UBNFDSXQBISC5QQSNGAAZIR7LRE7FCYRVYN5MMOY3A2ALEOQBYQQ, seg_pos={0 0}, upload=51329]: Long tail cancellation due to optimal threshold Dec 8 01:55:09.216585564 (58h26m9.087668609s) [seg_pos={0 0}, upload=51329]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 8 01:55:09.216654365 (58h26m9.0877374s) [upload=51329]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 01:55:09.216659064 (58h26m9.087742099s) [upload=51329]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 01:55:35.852213564 (58h26m35.723296629s) [upload=51329]: Done uploading segments... Dec 8 01:55:35.852273619 (58h26m35.723356664s) [upload=51329]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.613285499s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......X.........X..........................................] duration: 5.836838053s trace: https://jaeger.dev.storj.io/trace/e7164007ac0fd55 FAIL (2025-12-08 02:34:06 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.83691061s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [............X......X................X....X..................] duration: 3.413916209s trace: https://jaeger.dev.storj.io/trace/491130a80ad889c4 FAIL (2025-12-08 02:34:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.413943791s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 6.070414753s trace: https://jaeger.dev.storj.io/trace/27d92c196f473eec FAIL (2025-12-08 02:33:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 4891s) [upload=51469]: Got next segment. Inline: false Dec 8 02:04:21.143134547 (58h35m21.014217592s) [upload=51469]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 02:04:21.143141259 (58h35m21.014224304s) [upload=51469]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 02:04:22.267502706 (58h35m22.138585751s) [seg_pos={0 0}, upload=51469]: Begin upload segment... Dec 8 02:04:22.267624127 (58h35m22.138707172s) [seg_pos={0 0}, upload=51469]: Done begin upload segment. Dec 8 02:04:22.267698027 (58h35m22.138781073s) [noise=true , address=77.90.31.11:4554 , piece_id=BQP7XTRA2SBVINT4J7JAOF3ORNENBBIQN4ABMCEMQQ6FQL6QUOKA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.267712997 (58h35m22.138796042s) [noise=true , address=145.79.227.41:4087 , piece_id=XO6K3CXK7KZ5VEVXULHKQXLSSLIIWIX3VKQOSD7HY25WMNMCDLEQ, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.268434680 (58h35m22.139517715s) [noise=true , address=145.79.227.16:4910 , piece_id=ETE7ZBHT5IJUOBWCA5BGLKC2ANDEUB2M3DOMXWWS2RBWSJQBARRQ, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.268550331 (58h35m22.139633366s) [noise=true , address=145.79.227.26:4653 , piece_id=DKW72PEWPIOHPYHLI54PBU7NR3HGUBLFM3MN4XQQFWIIZ4FHIUNA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.268913111 (58h35m22.139996146s) [noise=true , address=145.79.227.23:4545 , piece_id=NBYCSU2KSHTAL223AHKHREQWKXUEQKJU46TY3SPYTKT5N5T2OUSA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.268924032 (58h35m22.140007067s) [noise=true , address=145.79.227.40:4051 , piece_id=7CQTGHFZ542WUHYX2CTOPN4PF2KDOOLDGV64THDBZCYOCXA3JWDA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.269398886 (58h35m22.140481921s) [noise=true , address=77.90.31.10:4518 , piece_id=6AR3NAFODP54SKG3SJSJSIQQFOYXO2C65WVNN4Z4XG426ZEOEDQA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.269414686 (58h35m22.140497721s) [noise=true , address=145.79.227.25:4619 , piece_id=XBNGVRIQLQUWKKBF4ULL7ZJDDT75QLY7ES2J7WHYWRD2ZLHOGU2Q, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.269753261 (58h35m22.140836296s) [noise=true , address=145.79.227.23:4547 , piece_id=5Y6V5QWLCWNOLT2GDNUEIRSG75KGNGIFLBCBGSIT7XH46RZYVDGQ, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.270016322 (58h35m22.141099367s) [noise=true , address=77.90.31.13:4628 , piece_id=LWTOCQRRA7HLI5ZPPRQ25IM22LMQNRR5H3TTOTSTGCFOYFZMABAA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.270259175 (58h35m22.14134222s) [noise=true , address=145.79.227.25:4617 , piece_id=S4ETCJ6WMCPXXQCYPPBBMHLMK2SVJV4S5VOKHGTDAGSGG6WRYH3Q, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.270655339 (58h35m22.141738374s) [noise=true , address=145.79.227.38:4977 , piece_id=73WXA33TWM4MFVTHDHW3PXJVABMCVMQYI3P42AHT3OYPKMXIVTUQ, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.271143158 (58h35m22.142246412s) [noise=true , address=145.79.227.28:4727 , piece_id=UYRK457JBF2AHCIXZ4XOIL7YHOVYZ7IPDUTB6WOZZUDEW5Q7NNCQ, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.271230565 (58h35m22.14231361s) [noise=true , address=145.79.227.39:4013 , piece_id=UI22RDXJ2DY3RFNSJVNT43XQJ6B2S3EO6SFLYHXW4VPVQJSYYF6Q, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.271662086 (58h35m22.142745121s) [noise=true , address=145.79.227.22:4511 , piece_id=GGA463DAOSZCW7SQGZW2LGFRQVNUOUJ2L2GXRXLSDPC65KVPY3TA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.272592658 (58h35m22.143675703s) [noise=true , address=145.79.227.24:4583 , piece_id=WYTXGXG77ZW4RGWX7YWDU2NH6R5CCPVBPQH3ISHUAJVKZJRONS5Q, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:24.009199123 (58h35m23.880282158s) [upload=51469]: Next returned nil segment Dec 8 02:04:24.009221235 (58h35m23.88030428s) [upload=51469]: Waiting for error group managing segments... Dec 8 02:04:24.293906896 (58h35m24.164989931s) [noise=true , address=145.79.227.24:4583 , piece_id=WYTXGXG77ZW4RGWX7YWDU2NH6R5CCPVBPQH3ISHUAJVKZJRONS5Q, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.298576738 (58h35m24.169659783s) [noise=true , address=145.79.227.28:4727 , piece_id=UYRK457JBF2AHCIXZ4XOIL7YHOVYZ7IPDUTB6WOZZUDEW5Q7NNCQ, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.301356571 (58h35m24.172439616s) [noise=true , address=145.79.227.25:4619 , piece_id=XBNGVRIQLQUWKKBF4ULL7ZJDDT75QLY7ES2J7WHYWRD2ZLHOGU2Q, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.326466765 (58h35m24.19754981s) [noise=true , address=145.79.227.16:4910 , piece_id=ETE7ZBHT5IJUOBWCA5BGLKC2ANDEUB2M3DOMXWWS2RBWSJQBARRQ, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.328633530 (58h35m24.199716586s) [noise=true , address=145.79.227.26:4653 , piece_id=DKW72PEWPIOHPYHLI54PBU7NR3HGUBLFM3MN4XQQFWIIZ4FHIUNA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.360508106 (58h35m24.231591151s) [noise=true , address=145.79.227.40:4051 , piece_id=7CQTGHFZ542WUHYX2CTOPN4PF2KDOOLDGV64THDBZCYOCXA3JWDA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.361116263 (58h35m24.232199308s) [noise=true , address=145.79.227.25:4617 , piece_id=S4ETCJ6WMCPXXQCYPPBBMHLMK2SVJV4S5VOKHGTDAGSGG6WRYH3Q, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.361695086 (58h35m24.232778131s) [noise=true , address=145.79.227.41:4087 , piece_id=XO6K3CXK7KZ5VEVXULHKQXLSSLIIWIX3VKQOSD7HY25WMNMCDLEQ, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.371520525 (58h35m24.24260356s) [noise=true , address=145.79.227.38:4977 , piece_id=73WXA33TWM4MFVTHDHW3PXJVABMCVMQYI3P42AHT3OYPKMXIVTUQ, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.381271893 (58h35m24.252354928s) [noise=true , address=145.79.227.23:4545 , piece_id=NBYCSU2KSHTAL223AHKHREQWKXUEQKJU46TY3SPYTKT5N5T2OUSA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.386403053 (58h35m24.257486098s) [noise=true , address=77.90.31.11:4554 , piece_id=BQP7XTRA2SBVINT4J7JAOF3ORNENBBIQN4ABMCEMQQ6FQL6QUOKA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.386427209 (58h35m24.257510254s) [noise=true , address=77.90.31.10:4518 , piece_id=6AR3NAFODP54SKG3SJSJSIQQFOYXO2C65WVNN4Z4XG426ZEOEDQA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.400489386 (58h35m24.271572431s) [noise=true , address=77.90.31.13:4628 , piece_id=LWTOCQRRA7HLI5ZPPRQ25IM22LMQNRR5H3TTOTSTGCFOYFZMABAA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.400513903 (58h35m24.271596938s) [seg_pos={0 0}, upload=51469]: Segment reached optimal threshold of 13 pieces. Dec 8 02:04:24.400813233 (58h35m24.271896288s) [noise=true , address=145.79.227.39:4013 , piece_id=UI22RDXJ2DY3RFNSJVNT43XQJ6B2S3EO6SFLYHXW4VPVQJSYYF6Q, seg_pos={0 0}, upload=51469]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b): protocol: write tcp 188.166.225.170:39434->145.79.227.39:4013: use of closed network connection; write tcp 188.166.225.170:39434->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39434->145.79.227.39:4013: use of closed network connection Dec 8 02:04:24.400825677 (58h35m24.271908722s) [noise=true , address=145.79.227.39:4013 , piece_id=UI22RDXJ2DY3RFNSJVNT43XQJ6B2S3EO6SFLYHXW4VPVQJSYYF6Q, seg_pos={0 0}, upload=51469]: Long tail cancellation due to optimal threshold Dec 8 02:04:24.400873027 (58h35m24.271956082s) [noise=true , address=145.79.227.22:4511 , piece_id=GGA463DAOSZCW7SQGZW2LGFRQVNUOUJ2L2GXRXLSDPC65KVPY3TA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG): protocol: write tcp 188.166.225.170:47926->145.79.227.22:4511: use of closed network connection; write tcp 188.166.225.170:47926->145.79.227.22:4511: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47926->145.79.227.22:4511: use of closed network connection Dec 8 02:04:24.400894248 (58h35m24.271977293s) [noise=true , address=145.79.227.22:4511 , piece_id=GGA463DAOSZCW7SQGZW2LGFRQVNUOUJ2L2GXRXLSDPC65KVPY3TA, seg_pos={0 0}, upload=51469]: Long tail cancellation due to optimal threshold Dec 8 02:04:24.401247129 (58h35m24.272330184s) [noise=true , address=145.79.227.23:4547 , piece_id=5Y6V5QWLCWNOLT2GDNUEIRSG75KGNGIFLBCBGSIT7XH46RZYVDGQ, seg_pos={0 0}, upload=51469]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW): protocol: read tcp 188.166.225.170:43344->145.79.227.23:4547: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 02:04:24.401260003 (58h35m24.272343059s) [noise=true , address=145.79.227.23:4547 , piece_id=5Y6V5QWLCWNOLT2GDNUEIRSG75KGNGIFLBCBGSIT7XH46RZYVDGQ, seg_pos={0 0}, upload=51469]: Long tail cancellation due to optimal threshold Dec 8 02:04:24.401285452 (58h35m24.272368497s) [seg_pos={0 0}, upload=51469]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 02:04:24.401382247 (58h35m24.272465292s) [upload=51469]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 02:04:24.401391394 (58h35m24.272474429s) [upload=51469]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 02:05:39.049922629 (58h36m38.921005704s) [upload=51469]: Done uploading segments... Dec 8 02:05:39.050003342 (58h36m38.921086377s) [upload=51469]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.070441904s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........X.........X.........................................] duration: 6.189238684s trace: https://jaeger.dev.storj.io/trace/e7164007ac0fd55 FAIL (2025-12-08 02:33:48 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 6.189275153s core uplink us1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 4.897290812s trace: https://jaeger.dev.storj.io/trace/7af096edea7ffcf5 FAIL (2025-12-08 02:33:48 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.897307503s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 6.273184622s trace: https://jaeger.dev.storj.io/trace/57258282a8e974fd FAIL (2025-12-08 02:33:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== UA4TAJ2FSYO5S4NXECVMMMCXKFL5CAHPQBQJ6WREQ3YFC3Q, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.202906305 (58h26m7.07398934s) [noise=true , address=45.91.116.4:4945 , piece_id=6UNWQQAWWWWC2XVUCDRCTAYJDTHBXAZJMXA6KLCTOP5PYSAJCCGA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.203371201 (58h26m7.074454246s) [noise=true , address=145.79.227.11:4740 , piece_id=KL54LL2UQRT6MH2K3NQQJB3BKPX4QRWQMXRK27MW2U7CHFDCCWFA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.203861354 (58h26m7.07494439s) [noise=true , address=77.90.31.12:4621 , piece_id=S4FVJHEPIO2MVO56ZRIJ27RMTV723MERPLJLVD2T2GAO3T7K24TA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.204290401 (58h26m7.075373446s) [noise=true , address=145.79.227.34:4928 , piece_id=DRKF2DRSCBAYSZO66A2ZON2JS6HIJIT2YUWFZRS3W2CKFTV3NLVA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.204711363 (58h26m7.075794398s) [noise=true , address=145.79.227.15:4908 , piece_id=JIUZSMOAKBFCFAX6RQ52TCNZMYBREQCYOEM75U3H2QYR4WH3KWCA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.205118208 (58h26m7.076201253s) [noise=true , address=77.90.31.18:4825 , piece_id=EJ4LO4TCEFB4OQQLFOEUYGQKK462ZPXXEIIY5CHF2PERXWDGQNDA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.205554277 (58h26m7.076637323s) [noise=true , address=145.79.227.30:4786 , piece_id=VM65RUN3GUZZFUOHFYE7ZLPWU6COCQUMY3IK7GCNWPIUZAMEBR7Q, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.206116137 (58h26m7.077199182s) [noise=true , address=145.79.227.25:4627 , piece_id=K7UYU3W2CACUAQUUL72GG5F7DP5CHAHZ4M47TZG6NBN5Z6ZCDRBA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.206664562 (58h26m7.077747607s) [noise=true , address=77.90.31.16:4760 , piece_id=Q67LYTDJCAR2TFVSAIIYCUYM7D44JFPBLCSKRNLXH26EAW5UQ3GA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.207102556 (58h26m7.078185601s) [noise=true , address=45.91.116.7:4037 , piece_id=UBNFDSXQBISC5QQSNGAAZIR7LRE7FCYRVYN5MMOY3A2ALEOQBYQQ, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.207519911 (58h26m7.078602956s) [noise=true , address=77.90.31.11:4579 , piece_id=JINRNEVUAP2VI5Y4CR7ALWDQIAIAFX3SP2VVZSXDDG6XPUKASE3Q, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.207967393 (58h26m7.079050438s) [noise=true , address=77.90.31.10:4550 , piece_id=PDCZ7VXNHQJ6LYPG64GZRYP4MRBGVPXCWTXSRT3UV2N6HGMDE2KA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.208416427 (58h26m7.079499472s) [noise=true , address=45.91.116.6:4003 , piece_id=CRONTV2B2AVZTLKU3YNIEGQZSON7XHHL7S3K6KNNJV4YR5FHQ3PA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.208830746 (58h26m7.079913791s) [noise=true , address=45.91.116.12:4254 , piece_id=UCKU4TBXJC4PSYHRDPUXETXLAILDOWA5NN7RCVVNYLVUBY7FBIFA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.209255245 (58h26m7.080338291s) [noise=true , address=145.79.227.16:4933 , piece_id=LHF64L3SWXFVZ5X5D67I44WSJNKRFB5G4YP7DFHDWOAYE23II3OA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.209662921 (58h26m7.080745966s) [noise=true , address=145.79.227.18:4010 , piece_id=S2EOTEV4TBNNDOBN7TIX76AJE2WI5P5DM3JK2NJB25TISYLS7VSQ, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.214493680 (58h26m7.085576725s) [noise=true , address=77.90.31.10:4539 , piece_id=27YFK55QLZDEULYABQ3OBDSPDDEIUJ2MJSD7X3A4K6HRFJMRQJUA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:08.555121654 (58h26m8.426204699s) [upload=51329]: Next returned nil segment Dec 8 01:55:08.555145519 (58h26m8.426228564s) [upload=51329]: Waiting for error group managing segments... Dec 8 01:55:08.935193756 (58h26m8.806276791s) [noise=true , address=64.49.13.27:4957 , piece_id=7METNUA4TAJ2FSYO5S4NXECVMMMCXKFL5CAHPQBQJ6WREQ3YFC3Q, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.093051314 (58h26m8.964134359s) [noise=true , address=145.79.227.16:4933 , piece_id=LHF64L3SWXFVZ5X5D67I44WSJNKRFB5G4YP7DFHDWOAYE23II3OA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.098812273 (58h26m8.969895318s) [noise=true , address=145.79.227.18:4010 , piece_id=S2EOTEV4TBNNDOBN7TIX76AJE2WI5P5DM3JK2NJB25TISYLS7VSQ, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.102616718 (58h26m8.973699763s) [noise=true , address=145.79.227.25:4627 , piece_id=K7UYU3W2CACUAQUUL72GG5F7DP5CHAHZ4M47TZG6NBN5Z6ZCDRBA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.118902949 (58h26m8.989985994s) [noise=true , address=145.79.227.34:4928 , piece_id=DRKF2DRSCBAYSZO66A2ZON2JS6HIJIT2YUWFZRS3W2CKFTV3NLVA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.124783838 (58h26m8.995866883s) [noise=true , address=77.90.31.18:4825 , piece_id=EJ4LO4TCEFB4OQQLFOEUYGQKK462ZPXXEIIY5CHF2PERXWDGQNDA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.126718282 (58h26m8.997801327s) [noise=true , address=145.79.227.11:4740 , piece_id=KL54LL2UQRT6MH2K3NQQJB3BKPX4QRWQMXRK27MW2U7CHFDCCWFA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.145158485 (58h26m9.01624153s) [noise=true , address=77.90.31.10:4550 , piece_id=PDCZ7VXNHQJ6LYPG64GZRYP4MRBGVPXCWTXSRT3UV2N6HGMDE2KA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.156650047 (58h26m9.027733092s) [noise=true , address=145.79.227.15:4908 , piece_id=JIUZSMOAKBFCFAX6RQ52TCNZMYBREQCYOEM75U3H2QYR4WH3KWCA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.162020363 (58h26m9.033103398s) [noise=true , address=77.90.31.10:4539 , piece_id=27YFK55QLZDEULYABQ3OBDSPDDEIUJ2MJSD7X3A4K6HRFJMRQJUA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.198899907 (58h26m9.069982942s) [noise=true , address=77.90.31.12:4621 , piece_id=S4FVJHEPIO2MVO56ZRIJ27RMTV723MERPLJLVD2T2GAO3T7K24TA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.208468337 (58h26m9.079551382s) [noise=true , address=45.91.116.4:4945 , piece_id=6UNWQQAWWWWC2XVUCDRCTAYJDTHBXAZJMXA6KLCTOP5PYSAJCCGA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.212217697 (58h26m9.083300732s) [noise=true , address=45.91.116.12:4254 , piece_id=UCKU4TBXJC4PSYHRDPUXETXLAILDOWA5NN7RCVVNYLVUBY7FBIFA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.215832390 (58h26m9.086915435s) [noise=true , address=77.90.31.11:4579 , piece_id=JINRNEVUAP2VI5Y4CR7ALWDQIAIAFX3SP2VVZSXDDG6XPUKASE3Q, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.215862086 (58h26m9.086945161s) [seg_pos={0 0}, upload=51329]: Segment reached optimal threshold of 14 pieces. Dec 8 01:55:09.216078098 (58h26m9.087161133s) [noise=true , address=45.91.116.6:4003 , piece_id=CRONTV2B2AVZTLKU3YNIEGQZSON7XHHL7S3K6KNNJV4YR5FHQ3PA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1veqoBGw6a8AGMoigyuNJMjcEvCT93xRq2B1d2KNhs6aEmSqVF): protocol: expected piece hash; context canceled; EOF Dec 8 01:55:09.216085662 (58h26m9.087168707s) [noise=true , address=45.91.116.6:4003 , piece_id=CRONTV2B2AVZTLKU3YNIEGQZSON7XHHL7S3K6KNNJV4YR5FHQ3PA, seg_pos={0 0}, upload=51329]: Long tail cancellation due to optimal threshold Dec 8 01:55:09.216307705 (58h26m9.08739075s) [noise=true , address=145.79.227.30:4786 , piece_id=VM65RUN3GUZZFUOHFYE7ZLPWU6COCQUMY3IK7GCNWPIUZAMEBR7Q, seg_pos={0 0}, upload=51329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125TWU7R4pkmrUxsNLhgTD475aLbjUJ7psA6drJtE7oDqZXYJ4Q): protocol: expected piece hash; context canceled; EOF Dec 8 01:55:09.216313956 (58h26m9.087396991s) [noise=true , address=145.79.227.30:4786 , piece_id=VM65RUN3GUZZFUOHFYE7ZLPWU6COCQUMY3IK7GCNWPIUZAMEBR7Q, seg_pos={0 0}, upload=51329]: Long tail cancellation due to optimal threshold Dec 8 01:55:09.216414809 (58h26m9.087497844s) [noise=true , address=77.90.31.16:4760 , piece_id=Q67LYTDJCAR2TFVSAIIYCUYM7D44JFPBLCSKRNLXH26EAW5UQ3GA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1e6p1TD5CqUyWdZjkrxh2kRqMuyDiMUr9eimfeayDAKwhtBrZ8): protocol: expected piece hash; context canceled; EOF Dec 8 01:55:09.216420681 (58h26m9.087503726s) [noise=true , address=77.90.31.16:4760 , piece_id=Q67LYTDJCAR2TFVSAIIYCUYM7D44JFPBLCSKRNLXH26EAW5UQ3GA, seg_pos={0 0}, upload=51329]: Long tail cancellation due to optimal threshold Dec 8 01:55:09.216562861 (58h26m9.087645906s) [noise=true , address=45.91.116.7:4037 , piece_id=UBNFDSXQBISC5QQSNGAAZIR7LRE7FCYRVYN5MMOY3A2ALEOQBYQQ, seg_pos={0 0}, upload=51329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iNPwE2XSfXcaXAXcrH2jcyNGYeDiiWvzAcDQffrmjPwTVqVmK): protocol: expected piece hash; context canceled; EOF Dec 8 01:55:09.216570025 (58h26m9.08765307s) [noise=true , address=45.91.116.7:4037 , piece_id=UBNFDSXQBISC5QQSNGAAZIR7LRE7FCYRVYN5MMOY3A2ALEOQBYQQ, seg_pos={0 0}, upload=51329]: Long tail cancellation due to optimal threshold Dec 8 01:55:09.216585564 (58h26m9.087668609s) [seg_pos={0 0}, upload=51329]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 8 01:55:09.216654365 (58h26m9.0877374s) [upload=51329]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 01:55:09.216659064 (58h26m9.087742099s) [upload=51329]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 01:55:35.852213564 (58h26m35.723296629s) [upload=51329]: Done uploading segments... Dec 8 01:55:35.852273619 (58h26m35.723356664s) [upload=51329]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.273236059s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 5.329686407s trace: https://jaeger.dev.storj.io/trace/49d209695fd399e8 FAIL (2025-12-08 02:33:45 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 5.329714451s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 6.055126963s trace: https://jaeger.dev.storj.io/trace/4030003299248e86 FAIL (2025-12-08 02:33:40 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.055156899s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 4.829815907s trace: https://jaeger.dev.storj.io/trace/7af096edea7ffcf5 FAIL (2025-12-08 02:33:24 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.829839803s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 5.872763451s trace: https://jaeger.dev.storj.io/trace/57258282a8e974fd FAIL (2025-12-08 02:33:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== UA4TAJ2FSYO5S4NXECVMMMCXKFL5CAHPQBQJ6WREQ3YFC3Q, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.202906305 (58h26m7.07398934s) [noise=true , address=45.91.116.4:4945 , piece_id=6UNWQQAWWWWC2XVUCDRCTAYJDTHBXAZJMXA6KLCTOP5PYSAJCCGA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.203371201 (58h26m7.074454246s) [noise=true , address=145.79.227.11:4740 , piece_id=KL54LL2UQRT6MH2K3NQQJB3BKPX4QRWQMXRK27MW2U7CHFDCCWFA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.203861354 (58h26m7.07494439s) [noise=true , address=77.90.31.12:4621 , piece_id=S4FVJHEPIO2MVO56ZRIJ27RMTV723MERPLJLVD2T2GAO3T7K24TA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.204290401 (58h26m7.075373446s) [noise=true , address=145.79.227.34:4928 , piece_id=DRKF2DRSCBAYSZO66A2ZON2JS6HIJIT2YUWFZRS3W2CKFTV3NLVA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.204711363 (58h26m7.075794398s) [noise=true , address=145.79.227.15:4908 , piece_id=JIUZSMOAKBFCFAX6RQ52TCNZMYBREQCYOEM75U3H2QYR4WH3KWCA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.205118208 (58h26m7.076201253s) [noise=true , address=77.90.31.18:4825 , piece_id=EJ4LO4TCEFB4OQQLFOEUYGQKK462ZPXXEIIY5CHF2PERXWDGQNDA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.205554277 (58h26m7.076637323s) [noise=true , address=145.79.227.30:4786 , piece_id=VM65RUN3GUZZFUOHFYE7ZLPWU6COCQUMY3IK7GCNWPIUZAMEBR7Q, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.206116137 (58h26m7.077199182s) [noise=true , address=145.79.227.25:4627 , piece_id=K7UYU3W2CACUAQUUL72GG5F7DP5CHAHZ4M47TZG6NBN5Z6ZCDRBA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.206664562 (58h26m7.077747607s) [noise=true , address=77.90.31.16:4760 , piece_id=Q67LYTDJCAR2TFVSAIIYCUYM7D44JFPBLCSKRNLXH26EAW5UQ3GA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.207102556 (58h26m7.078185601s) [noise=true , address=45.91.116.7:4037 , piece_id=UBNFDSXQBISC5QQSNGAAZIR7LRE7FCYRVYN5MMOY3A2ALEOQBYQQ, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.207519911 (58h26m7.078602956s) [noise=true , address=77.90.31.11:4579 , piece_id=JINRNEVUAP2VI5Y4CR7ALWDQIAIAFX3SP2VVZSXDDG6XPUKASE3Q, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.207967393 (58h26m7.079050438s) [noise=true , address=77.90.31.10:4550 , piece_id=PDCZ7VXNHQJ6LYPG64GZRYP4MRBGVPXCWTXSRT3UV2N6HGMDE2KA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.208416427 (58h26m7.079499472s) [noise=true , address=45.91.116.6:4003 , piece_id=CRONTV2B2AVZTLKU3YNIEGQZSON7XHHL7S3K6KNNJV4YR5FHQ3PA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.208830746 (58h26m7.079913791s) [noise=true , address=45.91.116.12:4254 , piece_id=UCKU4TBXJC4PSYHRDPUXETXLAILDOWA5NN7RCVVNYLVUBY7FBIFA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.209255245 (58h26m7.080338291s) [noise=true , address=145.79.227.16:4933 , piece_id=LHF64L3SWXFVZ5X5D67I44WSJNKRFB5G4YP7DFHDWOAYE23II3OA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.209662921 (58h26m7.080745966s) [noise=true , address=145.79.227.18:4010 , piece_id=S2EOTEV4TBNNDOBN7TIX76AJE2WI5P5DM3JK2NJB25TISYLS7VSQ, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.214493680 (58h26m7.085576725s) [noise=true , address=77.90.31.10:4539 , piece_id=27YFK55QLZDEULYABQ3OBDSPDDEIUJ2MJSD7X3A4K6HRFJMRQJUA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:08.555121654 (58h26m8.426204699s) [upload=51329]: Next returned nil segment Dec 8 01:55:08.555145519 (58h26m8.426228564s) [upload=51329]: Waiting for error group managing segments... Dec 8 01:55:08.935193756 (58h26m8.806276791s) [noise=true , address=64.49.13.27:4957 , piece_id=7METNUA4TAJ2FSYO5S4NXECVMMMCXKFL5CAHPQBQJ6WREQ3YFC3Q, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.093051314 (58h26m8.964134359s) [noise=true , address=145.79.227.16:4933 , piece_id=LHF64L3SWXFVZ5X5D67I44WSJNKRFB5G4YP7DFHDWOAYE23II3OA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.098812273 (58h26m8.969895318s) [noise=true , address=145.79.227.18:4010 , piece_id=S2EOTEV4TBNNDOBN7TIX76AJE2WI5P5DM3JK2NJB25TISYLS7VSQ, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.102616718 (58h26m8.973699763s) [noise=true , address=145.79.227.25:4627 , piece_id=K7UYU3W2CACUAQUUL72GG5F7DP5CHAHZ4M47TZG6NBN5Z6ZCDRBA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.118902949 (58h26m8.989985994s) [noise=true , address=145.79.227.34:4928 , piece_id=DRKF2DRSCBAYSZO66A2ZON2JS6HIJIT2YUWFZRS3W2CKFTV3NLVA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.124783838 (58h26m8.995866883s) [noise=true , address=77.90.31.18:4825 , piece_id=EJ4LO4TCEFB4OQQLFOEUYGQKK462ZPXXEIIY5CHF2PERXWDGQNDA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.126718282 (58h26m8.997801327s) [noise=true , address=145.79.227.11:4740 , piece_id=KL54LL2UQRT6MH2K3NQQJB3BKPX4QRWQMXRK27MW2U7CHFDCCWFA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.145158485 (58h26m9.01624153s) [noise=true , address=77.90.31.10:4550 , piece_id=PDCZ7VXNHQJ6LYPG64GZRYP4MRBGVPXCWTXSRT3UV2N6HGMDE2KA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.156650047 (58h26m9.027733092s) [noise=true , address=145.79.227.15:4908 , piece_id=JIUZSMOAKBFCFAX6RQ52TCNZMYBREQCYOEM75U3H2QYR4WH3KWCA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.162020363 (58h26m9.033103398s) [noise=true , address=77.90.31.10:4539 , piece_id=27YFK55QLZDEULYABQ3OBDSPDDEIUJ2MJSD7X3A4K6HRFJMRQJUA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.198899907 (58h26m9.069982942s) [noise=true , address=77.90.31.12:4621 , piece_id=S4FVJHEPIO2MVO56ZRIJ27RMTV723MERPLJLVD2T2GAO3T7K24TA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.208468337 (58h26m9.079551382s) [noise=true , address=45.91.116.4:4945 , piece_id=6UNWQQAWWWWC2XVUCDRCTAYJDTHBXAZJMXA6KLCTOP5PYSAJCCGA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.212217697 (58h26m9.083300732s) [noise=true , address=45.91.116.12:4254 , piece_id=UCKU4TBXJC4PSYHRDPUXETXLAILDOWA5NN7RCVVNYLVUBY7FBIFA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.215832390 (58h26m9.086915435s) [noise=true , address=77.90.31.11:4579 , piece_id=JINRNEVUAP2VI5Y4CR7ALWDQIAIAFX3SP2VVZSXDDG6XPUKASE3Q, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.215862086 (58h26m9.086945161s) [seg_pos={0 0}, upload=51329]: Segment reached optimal threshold of 14 pieces. Dec 8 01:55:09.216078098 (58h26m9.087161133s) [noise=true , address=45.91.116.6:4003 , piece_id=CRONTV2B2AVZTLKU3YNIEGQZSON7XHHL7S3K6KNNJV4YR5FHQ3PA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1veqoBGw6a8AGMoigyuNJMjcEvCT93xRq2B1d2KNhs6aEmSqVF): protocol: expected piece hash; context canceled; EOF Dec 8 01:55:09.216085662 (58h26m9.087168707s) [noise=true , address=45.91.116.6:4003 , piece_id=CRONTV2B2AVZTLKU3YNIEGQZSON7XHHL7S3K6KNNJV4YR5FHQ3PA, seg_pos={0 0}, upload=51329]: Long tail cancellation due to optimal threshold Dec 8 01:55:09.216307705 (58h26m9.08739075s) [noise=true , address=145.79.227.30:4786 , piece_id=VM65RUN3GUZZFUOHFYE7ZLPWU6COCQUMY3IK7GCNWPIUZAMEBR7Q, seg_pos={0 0}, upload=51329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125TWU7R4pkmrUxsNLhgTD475aLbjUJ7psA6drJtE7oDqZXYJ4Q): protocol: expected piece hash; context canceled; EOF Dec 8 01:55:09.216313956 (58h26m9.087396991s) [noise=true , address=145.79.227.30:4786 , piece_id=VM65RUN3GUZZFUOHFYE7ZLPWU6COCQUMY3IK7GCNWPIUZAMEBR7Q, seg_pos={0 0}, upload=51329]: Long tail cancellation due to optimal threshold Dec 8 01:55:09.216414809 (58h26m9.087497844s) [noise=true , address=77.90.31.16:4760 , piece_id=Q67LYTDJCAR2TFVSAIIYCUYM7D44JFPBLCSKRNLXH26EAW5UQ3GA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1e6p1TD5CqUyWdZjkrxh2kRqMuyDiMUr9eimfeayDAKwhtBrZ8): protocol: expected piece hash; context canceled; EOF Dec 8 01:55:09.216420681 (58h26m9.087503726s) [noise=true , address=77.90.31.16:4760 , piece_id=Q67LYTDJCAR2TFVSAIIYCUYM7D44JFPBLCSKRNLXH26EAW5UQ3GA, seg_pos={0 0}, upload=51329]: Long tail cancellation due to optimal threshold Dec 8 01:55:09.216562861 (58h26m9.087645906s) [noise=true , address=45.91.116.7:4037 , piece_id=UBNFDSXQBISC5QQSNGAAZIR7LRE7FCYRVYN5MMOY3A2ALEOQBYQQ, seg_pos={0 0}, upload=51329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iNPwE2XSfXcaXAXcrH2jcyNGYeDiiWvzAcDQffrmjPwTVqVmK): protocol: expected piece hash; context canceled; EOF Dec 8 01:55:09.216570025 (58h26m9.08765307s) [noise=true , address=45.91.116.7:4037 , piece_id=UBNFDSXQBISC5QQSNGAAZIR7LRE7FCYRVYN5MMOY3A2ALEOQBYQQ, seg_pos={0 0}, upload=51329]: Long tail cancellation due to optimal threshold Dec 8 01:55:09.216585564 (58h26m9.087668609s) [seg_pos={0 0}, upload=51329]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 8 01:55:09.216654365 (58h26m9.0877374s) [upload=51329]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 01:55:09.216659064 (58h26m9.087742099s) [upload=51329]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 01:55:35.852213564 (58h26m35.723296629s) [upload=51329]: Done uploading segments... Dec 8 01:55:35.852273619 (58h26m35.723356664s) [upload=51329]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.872796854s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 5.615749883s trace: https://jaeger.dev.storj.io/trace/49d209695fd399e8 FAIL (2025-12-08 02:33:19 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 5.615768848s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 7.329217201s trace: https://jaeger.dev.storj.io/trace/27d92c196f473eec FAIL (2025-12-08 02:33:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 4891s) [upload=51469]: Got next segment. Inline: false Dec 8 02:04:21.143134547 (58h35m21.014217592s) [upload=51469]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 02:04:21.143141259 (58h35m21.014224304s) [upload=51469]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 02:04:22.267502706 (58h35m22.138585751s) [seg_pos={0 0}, upload=51469]: Begin upload segment... Dec 8 02:04:22.267624127 (58h35m22.138707172s) [seg_pos={0 0}, upload=51469]: Done begin upload segment. Dec 8 02:04:22.267698027 (58h35m22.138781073s) [noise=true , address=77.90.31.11:4554 , piece_id=BQP7XTRA2SBVINT4J7JAOF3ORNENBBIQN4ABMCEMQQ6FQL6QUOKA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.267712997 (58h35m22.138796042s) [noise=true , address=145.79.227.41:4087 , piece_id=XO6K3CXK7KZ5VEVXULHKQXLSSLIIWIX3VKQOSD7HY25WMNMCDLEQ, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.268434680 (58h35m22.139517715s) [noise=true , address=145.79.227.16:4910 , piece_id=ETE7ZBHT5IJUOBWCA5BGLKC2ANDEUB2M3DOMXWWS2RBWSJQBARRQ, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.268550331 (58h35m22.139633366s) [noise=true , address=145.79.227.26:4653 , piece_id=DKW72PEWPIOHPYHLI54PBU7NR3HGUBLFM3MN4XQQFWIIZ4FHIUNA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.268913111 (58h35m22.139996146s) [noise=true , address=145.79.227.23:4545 , piece_id=NBYCSU2KSHTAL223AHKHREQWKXUEQKJU46TY3SPYTKT5N5T2OUSA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.268924032 (58h35m22.140007067s) [noise=true , address=145.79.227.40:4051 , piece_id=7CQTGHFZ542WUHYX2CTOPN4PF2KDOOLDGV64THDBZCYOCXA3JWDA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.269398886 (58h35m22.140481921s) [noise=true , address=77.90.31.10:4518 , piece_id=6AR3NAFODP54SKG3SJSJSIQQFOYXO2C65WVNN4Z4XG426ZEOEDQA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.269414686 (58h35m22.140497721s) [noise=true , address=145.79.227.25:4619 , piece_id=XBNGVRIQLQUWKKBF4ULL7ZJDDT75QLY7ES2J7WHYWRD2ZLHOGU2Q, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.269753261 (58h35m22.140836296s) [noise=true , address=145.79.227.23:4547 , piece_id=5Y6V5QWLCWNOLT2GDNUEIRSG75KGNGIFLBCBGSIT7XH46RZYVDGQ, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.270016322 (58h35m22.141099367s) [noise=true , address=77.90.31.13:4628 , piece_id=LWTOCQRRA7HLI5ZPPRQ25IM22LMQNRR5H3TTOTSTGCFOYFZMABAA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.270259175 (58h35m22.14134222s) [noise=true , address=145.79.227.25:4617 , piece_id=S4ETCJ6WMCPXXQCYPPBBMHLMK2SVJV4S5VOKHGTDAGSGG6WRYH3Q, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.270655339 (58h35m22.141738374s) [noise=true , address=145.79.227.38:4977 , piece_id=73WXA33TWM4MFVTHDHW3PXJVABMCVMQYI3P42AHT3OYPKMXIVTUQ, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.271143158 (58h35m22.142246412s) [noise=true , address=145.79.227.28:4727 , piece_id=UYRK457JBF2AHCIXZ4XOIL7YHOVYZ7IPDUTB6WOZZUDEW5Q7NNCQ, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.271230565 (58h35m22.14231361s) [noise=true , address=145.79.227.39:4013 , piece_id=UI22RDXJ2DY3RFNSJVNT43XQJ6B2S3EO6SFLYHXW4VPVQJSYYF6Q, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.271662086 (58h35m22.142745121s) [noise=true , address=145.79.227.22:4511 , piece_id=GGA463DAOSZCW7SQGZW2LGFRQVNUOUJ2L2GXRXLSDPC65KVPY3TA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.272592658 (58h35m22.143675703s) [noise=true , address=145.79.227.24:4583 , piece_id=WYTXGXG77ZW4RGWX7YWDU2NH6R5CCPVBPQH3ISHUAJVKZJRONS5Q, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:24.009199123 (58h35m23.880282158s) [upload=51469]: Next returned nil segment Dec 8 02:04:24.009221235 (58h35m23.88030428s) [upload=51469]: Waiting for error group managing segments... Dec 8 02:04:24.293906896 (58h35m24.164989931s) [noise=true , address=145.79.227.24:4583 , piece_id=WYTXGXG77ZW4RGWX7YWDU2NH6R5CCPVBPQH3ISHUAJVKZJRONS5Q, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.298576738 (58h35m24.169659783s) [noise=true , address=145.79.227.28:4727 , piece_id=UYRK457JBF2AHCIXZ4XOIL7YHOVYZ7IPDUTB6WOZZUDEW5Q7NNCQ, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.301356571 (58h35m24.172439616s) [noise=true , address=145.79.227.25:4619 , piece_id=XBNGVRIQLQUWKKBF4ULL7ZJDDT75QLY7ES2J7WHYWRD2ZLHOGU2Q, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.326466765 (58h35m24.19754981s) [noise=true , address=145.79.227.16:4910 , piece_id=ETE7ZBHT5IJUOBWCA5BGLKC2ANDEUB2M3DOMXWWS2RBWSJQBARRQ, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.328633530 (58h35m24.199716586s) [noise=true , address=145.79.227.26:4653 , piece_id=DKW72PEWPIOHPYHLI54PBU7NR3HGUBLFM3MN4XQQFWIIZ4FHIUNA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.360508106 (58h35m24.231591151s) [noise=true , address=145.79.227.40:4051 , piece_id=7CQTGHFZ542WUHYX2CTOPN4PF2KDOOLDGV64THDBZCYOCXA3JWDA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.361116263 (58h35m24.232199308s) [noise=true , address=145.79.227.25:4617 , piece_id=S4ETCJ6WMCPXXQCYPPBBMHLMK2SVJV4S5VOKHGTDAGSGG6WRYH3Q, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.361695086 (58h35m24.232778131s) [noise=true , address=145.79.227.41:4087 , piece_id=XO6K3CXK7KZ5VEVXULHKQXLSSLIIWIX3VKQOSD7HY25WMNMCDLEQ, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.371520525 (58h35m24.24260356s) [noise=true , address=145.79.227.38:4977 , piece_id=73WXA33TWM4MFVTHDHW3PXJVABMCVMQYI3P42AHT3OYPKMXIVTUQ, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.381271893 (58h35m24.252354928s) [noise=true , address=145.79.227.23:4545 , piece_id=NBYCSU2KSHTAL223AHKHREQWKXUEQKJU46TY3SPYTKT5N5T2OUSA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.386403053 (58h35m24.257486098s) [noise=true , address=77.90.31.11:4554 , piece_id=BQP7XTRA2SBVINT4J7JAOF3ORNENBBIQN4ABMCEMQQ6FQL6QUOKA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.386427209 (58h35m24.257510254s) [noise=true , address=77.90.31.10:4518 , piece_id=6AR3NAFODP54SKG3SJSJSIQQFOYXO2C65WVNN4Z4XG426ZEOEDQA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.400489386 (58h35m24.271572431s) [noise=true , address=77.90.31.13:4628 , piece_id=LWTOCQRRA7HLI5ZPPRQ25IM22LMQNRR5H3TTOTSTGCFOYFZMABAA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.400513903 (58h35m24.271596938s) [seg_pos={0 0}, upload=51469]: Segment reached optimal threshold of 13 pieces. Dec 8 02:04:24.400813233 (58h35m24.271896288s) [noise=true , address=145.79.227.39:4013 , piece_id=UI22RDXJ2DY3RFNSJVNT43XQJ6B2S3EO6SFLYHXW4VPVQJSYYF6Q, seg_pos={0 0}, upload=51469]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b): protocol: write tcp 188.166.225.170:39434->145.79.227.39:4013: use of closed network connection; write tcp 188.166.225.170:39434->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39434->145.79.227.39:4013: use of closed network connection Dec 8 02:04:24.400825677 (58h35m24.271908722s) [noise=true , address=145.79.227.39:4013 , piece_id=UI22RDXJ2DY3RFNSJVNT43XQJ6B2S3EO6SFLYHXW4VPVQJSYYF6Q, seg_pos={0 0}, upload=51469]: Long tail cancellation due to optimal threshold Dec 8 02:04:24.400873027 (58h35m24.271956082s) [noise=true , address=145.79.227.22:4511 , piece_id=GGA463DAOSZCW7SQGZW2LGFRQVNUOUJ2L2GXRXLSDPC65KVPY3TA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG): protocol: write tcp 188.166.225.170:47926->145.79.227.22:4511: use of closed network connection; write tcp 188.166.225.170:47926->145.79.227.22:4511: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47926->145.79.227.22:4511: use of closed network connection Dec 8 02:04:24.400894248 (58h35m24.271977293s) [noise=true , address=145.79.227.22:4511 , piece_id=GGA463DAOSZCW7SQGZW2LGFRQVNUOUJ2L2GXRXLSDPC65KVPY3TA, seg_pos={0 0}, upload=51469]: Long tail cancellation due to optimal threshold Dec 8 02:04:24.401247129 (58h35m24.272330184s) [noise=true , address=145.79.227.23:4547 , piece_id=5Y6V5QWLCWNOLT2GDNUEIRSG75KGNGIFLBCBGSIT7XH46RZYVDGQ, seg_pos={0 0}, upload=51469]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW): protocol: read tcp 188.166.225.170:43344->145.79.227.23:4547: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 02:04:24.401260003 (58h35m24.272343059s) [noise=true , address=145.79.227.23:4547 , piece_id=5Y6V5QWLCWNOLT2GDNUEIRSG75KGNGIFLBCBGSIT7XH46RZYVDGQ, seg_pos={0 0}, upload=51469]: Long tail cancellation due to optimal threshold Dec 8 02:04:24.401285452 (58h35m24.272368497s) [seg_pos={0 0}, upload=51469]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 02:04:24.401382247 (58h35m24.272465292s) [upload=51469]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 02:04:24.401391394 (58h35m24.272474429s) [upload=51469]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 02:05:39.049922629 (58h36m38.921005704s) [upload=51469]: Done uploading segments... Dec 8 02:05:39.050003342 (58h36m38.921086377s) [upload=51469]: Upload finished. err: metaclient: context deadline exceeded Duration: 7.329246038s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 5.728398394s trace: https://jaeger.dev.storj.io/trace/4030003299248e86 FAIL (2025-12-08 02:33:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.728423624s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [.............X......X................X....X.................] duration: 3.623885677s trace: https://jaeger.dev.storj.io/trace/491130a80ad889c4 FAIL (2025-12-08 02:32:53 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.623913791s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 5.989186461s trace: https://jaeger.dev.storj.io/trace/57258282a8e974fd FAIL (2025-12-08 02:32:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== UA4TAJ2FSYO5S4NXECVMMMCXKFL5CAHPQBQJ6WREQ3YFC3Q, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.202906305 (58h26m7.07398934s) [noise=true , address=45.91.116.4:4945 , piece_id=6UNWQQAWWWWC2XVUCDRCTAYJDTHBXAZJMXA6KLCTOP5PYSAJCCGA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.203371201 (58h26m7.074454246s) [noise=true , address=145.79.227.11:4740 , piece_id=KL54LL2UQRT6MH2K3NQQJB3BKPX4QRWQMXRK27MW2U7CHFDCCWFA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.203861354 (58h26m7.07494439s) [noise=true , address=77.90.31.12:4621 , piece_id=S4FVJHEPIO2MVO56ZRIJ27RMTV723MERPLJLVD2T2GAO3T7K24TA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.204290401 (58h26m7.075373446s) [noise=true , address=145.79.227.34:4928 , piece_id=DRKF2DRSCBAYSZO66A2ZON2JS6HIJIT2YUWFZRS3W2CKFTV3NLVA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.204711363 (58h26m7.075794398s) [noise=true , address=145.79.227.15:4908 , piece_id=JIUZSMOAKBFCFAX6RQ52TCNZMYBREQCYOEM75U3H2QYR4WH3KWCA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.205118208 (58h26m7.076201253s) [noise=true , address=77.90.31.18:4825 , piece_id=EJ4LO4TCEFB4OQQLFOEUYGQKK462ZPXXEIIY5CHF2PERXWDGQNDA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.205554277 (58h26m7.076637323s) [noise=true , address=145.79.227.30:4786 , piece_id=VM65RUN3GUZZFUOHFYE7ZLPWU6COCQUMY3IK7GCNWPIUZAMEBR7Q, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.206116137 (58h26m7.077199182s) [noise=true , address=145.79.227.25:4627 , piece_id=K7UYU3W2CACUAQUUL72GG5F7DP5CHAHZ4M47TZG6NBN5Z6ZCDRBA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.206664562 (58h26m7.077747607s) [noise=true , address=77.90.31.16:4760 , piece_id=Q67LYTDJCAR2TFVSAIIYCUYM7D44JFPBLCSKRNLXH26EAW5UQ3GA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.207102556 (58h26m7.078185601s) [noise=true , address=45.91.116.7:4037 , piece_id=UBNFDSXQBISC5QQSNGAAZIR7LRE7FCYRVYN5MMOY3A2ALEOQBYQQ, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.207519911 (58h26m7.078602956s) [noise=true , address=77.90.31.11:4579 , piece_id=JINRNEVUAP2VI5Y4CR7ALWDQIAIAFX3SP2VVZSXDDG6XPUKASE3Q, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.207967393 (58h26m7.079050438s) [noise=true , address=77.90.31.10:4550 , piece_id=PDCZ7VXNHQJ6LYPG64GZRYP4MRBGVPXCWTXSRT3UV2N6HGMDE2KA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.208416427 (58h26m7.079499472s) [noise=true , address=45.91.116.6:4003 , piece_id=CRONTV2B2AVZTLKU3YNIEGQZSON7XHHL7S3K6KNNJV4YR5FHQ3PA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.208830746 (58h26m7.079913791s) [noise=true , address=45.91.116.12:4254 , piece_id=UCKU4TBXJC4PSYHRDPUXETXLAILDOWA5NN7RCVVNYLVUBY7FBIFA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.209255245 (58h26m7.080338291s) [noise=true , address=145.79.227.16:4933 , piece_id=LHF64L3SWXFVZ5X5D67I44WSJNKRFB5G4YP7DFHDWOAYE23II3OA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.209662921 (58h26m7.080745966s) [noise=true , address=145.79.227.18:4010 , piece_id=S2EOTEV4TBNNDOBN7TIX76AJE2WI5P5DM3JK2NJB25TISYLS7VSQ, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.214493680 (58h26m7.085576725s) [noise=true , address=77.90.31.10:4539 , piece_id=27YFK55QLZDEULYABQ3OBDSPDDEIUJ2MJSD7X3A4K6HRFJMRQJUA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:08.555121654 (58h26m8.426204699s) [upload=51329]: Next returned nil segment Dec 8 01:55:08.555145519 (58h26m8.426228564s) [upload=51329]: Waiting for error group managing segments... Dec 8 01:55:08.935193756 (58h26m8.806276791s) [noise=true , address=64.49.13.27:4957 , piece_id=7METNUA4TAJ2FSYO5S4NXECVMMMCXKFL5CAHPQBQJ6WREQ3YFC3Q, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.093051314 (58h26m8.964134359s) [noise=true , address=145.79.227.16:4933 , piece_id=LHF64L3SWXFVZ5X5D67I44WSJNKRFB5G4YP7DFHDWOAYE23II3OA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.098812273 (58h26m8.969895318s) [noise=true , address=145.79.227.18:4010 , piece_id=S2EOTEV4TBNNDOBN7TIX76AJE2WI5P5DM3JK2NJB25TISYLS7VSQ, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.102616718 (58h26m8.973699763s) [noise=true , address=145.79.227.25:4627 , piece_id=K7UYU3W2CACUAQUUL72GG5F7DP5CHAHZ4M47TZG6NBN5Z6ZCDRBA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.118902949 (58h26m8.989985994s) [noise=true , address=145.79.227.34:4928 , piece_id=DRKF2DRSCBAYSZO66A2ZON2JS6HIJIT2YUWFZRS3W2CKFTV3NLVA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.124783838 (58h26m8.995866883s) [noise=true , address=77.90.31.18:4825 , piece_id=EJ4LO4TCEFB4OQQLFOEUYGQKK462ZPXXEIIY5CHF2PERXWDGQNDA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.126718282 (58h26m8.997801327s) [noise=true , address=145.79.227.11:4740 , piece_id=KL54LL2UQRT6MH2K3NQQJB3BKPX4QRWQMXRK27MW2U7CHFDCCWFA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.145158485 (58h26m9.01624153s) [noise=true , address=77.90.31.10:4550 , piece_id=PDCZ7VXNHQJ6LYPG64GZRYP4MRBGVPXCWTXSRT3UV2N6HGMDE2KA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.156650047 (58h26m9.027733092s) [noise=true , address=145.79.227.15:4908 , piece_id=JIUZSMOAKBFCFAX6RQ52TCNZMYBREQCYOEM75U3H2QYR4WH3KWCA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.162020363 (58h26m9.033103398s) [noise=true , address=77.90.31.10:4539 , piece_id=27YFK55QLZDEULYABQ3OBDSPDDEIUJ2MJSD7X3A4K6HRFJMRQJUA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.198899907 (58h26m9.069982942s) [noise=true , address=77.90.31.12:4621 , piece_id=S4FVJHEPIO2MVO56ZRIJ27RMTV723MERPLJLVD2T2GAO3T7K24TA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.208468337 (58h26m9.079551382s) [noise=true , address=45.91.116.4:4945 , piece_id=6UNWQQAWWWWC2XVUCDRCTAYJDTHBXAZJMXA6KLCTOP5PYSAJCCGA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.212217697 (58h26m9.083300732s) [noise=true , address=45.91.116.12:4254 , piece_id=UCKU4TBXJC4PSYHRDPUXETXLAILDOWA5NN7RCVVNYLVUBY7FBIFA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.215832390 (58h26m9.086915435s) [noise=true , address=77.90.31.11:4579 , piece_id=JINRNEVUAP2VI5Y4CR7ALWDQIAIAFX3SP2VVZSXDDG6XPUKASE3Q, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.215862086 (58h26m9.086945161s) [seg_pos={0 0}, upload=51329]: Segment reached optimal threshold of 14 pieces. Dec 8 01:55:09.216078098 (58h26m9.087161133s) [noise=true , address=45.91.116.6:4003 , piece_id=CRONTV2B2AVZTLKU3YNIEGQZSON7XHHL7S3K6KNNJV4YR5FHQ3PA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1veqoBGw6a8AGMoigyuNJMjcEvCT93xRq2B1d2KNhs6aEmSqVF): protocol: expected piece hash; context canceled; EOF Dec 8 01:55:09.216085662 (58h26m9.087168707s) [noise=true , address=45.91.116.6:4003 , piece_id=CRONTV2B2AVZTLKU3YNIEGQZSON7XHHL7S3K6KNNJV4YR5FHQ3PA, seg_pos={0 0}, upload=51329]: Long tail cancellation due to optimal threshold Dec 8 01:55:09.216307705 (58h26m9.08739075s) [noise=true , address=145.79.227.30:4786 , piece_id=VM65RUN3GUZZFUOHFYE7ZLPWU6COCQUMY3IK7GCNWPIUZAMEBR7Q, seg_pos={0 0}, upload=51329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125TWU7R4pkmrUxsNLhgTD475aLbjUJ7psA6drJtE7oDqZXYJ4Q): protocol: expected piece hash; context canceled; EOF Dec 8 01:55:09.216313956 (58h26m9.087396991s) [noise=true , address=145.79.227.30:4786 , piece_id=VM65RUN3GUZZFUOHFYE7ZLPWU6COCQUMY3IK7GCNWPIUZAMEBR7Q, seg_pos={0 0}, upload=51329]: Long tail cancellation due to optimal threshold Dec 8 01:55:09.216414809 (58h26m9.087497844s) [noise=true , address=77.90.31.16:4760 , piece_id=Q67LYTDJCAR2TFVSAIIYCUYM7D44JFPBLCSKRNLXH26EAW5UQ3GA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1e6p1TD5CqUyWdZjkrxh2kRqMuyDiMUr9eimfeayDAKwhtBrZ8): protocol: expected piece hash; context canceled; EOF Dec 8 01:55:09.216420681 (58h26m9.087503726s) [noise=true , address=77.90.31.16:4760 , piece_id=Q67LYTDJCAR2TFVSAIIYCUYM7D44JFPBLCSKRNLXH26EAW5UQ3GA, seg_pos={0 0}, upload=51329]: Long tail cancellation due to optimal threshold Dec 8 01:55:09.216562861 (58h26m9.087645906s) [noise=true , address=45.91.116.7:4037 , piece_id=UBNFDSXQBISC5QQSNGAAZIR7LRE7FCYRVYN5MMOY3A2ALEOQBYQQ, seg_pos={0 0}, upload=51329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iNPwE2XSfXcaXAXcrH2jcyNGYeDiiWvzAcDQffrmjPwTVqVmK): protocol: expected piece hash; context canceled; EOF Dec 8 01:55:09.216570025 (58h26m9.08765307s) [noise=true , address=45.91.116.7:4037 , piece_id=UBNFDSXQBISC5QQSNGAAZIR7LRE7FCYRVYN5MMOY3A2ALEOQBYQQ, seg_pos={0 0}, upload=51329]: Long tail cancellation due to optimal threshold Dec 8 01:55:09.216585564 (58h26m9.087668609s) [seg_pos={0 0}, upload=51329]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 8 01:55:09.216654365 (58h26m9.0877374s) [upload=51329]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 01:55:09.216659064 (58h26m9.087742099s) [upload=51329]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 01:55:35.852213564 (58h26m35.723296629s) [upload=51329]: Done uploading segments... Dec 8 01:55:35.852273619 (58h26m35.723356664s) [upload=51329]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.989206599s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........X.........X........................................] duration: 5.847060988s trace: https://jaeger.dev.storj.io/trace/e7164007ac0fd55 FAIL (2025-12-08 02:32:48 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.847090246s core uplink us1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 17.486841902s trace: https://jaeger.dev.storj.io/trace/49d209695fd399e8 FAIL (2025-12-08 02:32:41 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 17.486870707s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 4.926550957s trace: https://jaeger.dev.storj.io/trace/7af096edea7ffcf5 FAIL (2025-12-08 02:32:23 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.926585844s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 5.74026507s trace: https://jaeger.dev.storj.io/trace/57258282a8e974fd FAIL (2025-12-08 02:32:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== UA4TAJ2FSYO5S4NXECVMMMCXKFL5CAHPQBQJ6WREQ3YFC3Q, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.202906305 (58h26m7.07398934s) [noise=true , address=45.91.116.4:4945 , piece_id=6UNWQQAWWWWC2XVUCDRCTAYJDTHBXAZJMXA6KLCTOP5PYSAJCCGA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.203371201 (58h26m7.074454246s) [noise=true , address=145.79.227.11:4740 , piece_id=KL54LL2UQRT6MH2K3NQQJB3BKPX4QRWQMXRK27MW2U7CHFDCCWFA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.203861354 (58h26m7.07494439s) [noise=true , address=77.90.31.12:4621 , piece_id=S4FVJHEPIO2MVO56ZRIJ27RMTV723MERPLJLVD2T2GAO3T7K24TA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.204290401 (58h26m7.075373446s) [noise=true , address=145.79.227.34:4928 , piece_id=DRKF2DRSCBAYSZO66A2ZON2JS6HIJIT2YUWFZRS3W2CKFTV3NLVA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.204711363 (58h26m7.075794398s) [noise=true , address=145.79.227.15:4908 , piece_id=JIUZSMOAKBFCFAX6RQ52TCNZMYBREQCYOEM75U3H2QYR4WH3KWCA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.205118208 (58h26m7.076201253s) [noise=true , address=77.90.31.18:4825 , piece_id=EJ4LO4TCEFB4OQQLFOEUYGQKK462ZPXXEIIY5CHF2PERXWDGQNDA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.205554277 (58h26m7.076637323s) [noise=true , address=145.79.227.30:4786 , piece_id=VM65RUN3GUZZFUOHFYE7ZLPWU6COCQUMY3IK7GCNWPIUZAMEBR7Q, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.206116137 (58h26m7.077199182s) [noise=true , address=145.79.227.25:4627 , piece_id=K7UYU3W2CACUAQUUL72GG5F7DP5CHAHZ4M47TZG6NBN5Z6ZCDRBA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.206664562 (58h26m7.077747607s) [noise=true , address=77.90.31.16:4760 , piece_id=Q67LYTDJCAR2TFVSAIIYCUYM7D44JFPBLCSKRNLXH26EAW5UQ3GA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.207102556 (58h26m7.078185601s) [noise=true , address=45.91.116.7:4037 , piece_id=UBNFDSXQBISC5QQSNGAAZIR7LRE7FCYRVYN5MMOY3A2ALEOQBYQQ, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.207519911 (58h26m7.078602956s) [noise=true , address=77.90.31.11:4579 , piece_id=JINRNEVUAP2VI5Y4CR7ALWDQIAIAFX3SP2VVZSXDDG6XPUKASE3Q, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.207967393 (58h26m7.079050438s) [noise=true , address=77.90.31.10:4550 , piece_id=PDCZ7VXNHQJ6LYPG64GZRYP4MRBGVPXCWTXSRT3UV2N6HGMDE2KA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.208416427 (58h26m7.079499472s) [noise=true , address=45.91.116.6:4003 , piece_id=CRONTV2B2AVZTLKU3YNIEGQZSON7XHHL7S3K6KNNJV4YR5FHQ3PA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.208830746 (58h26m7.079913791s) [noise=true , address=45.91.116.12:4254 , piece_id=UCKU4TBXJC4PSYHRDPUXETXLAILDOWA5NN7RCVVNYLVUBY7FBIFA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.209255245 (58h26m7.080338291s) [noise=true , address=145.79.227.16:4933 , piece_id=LHF64L3SWXFVZ5X5D67I44WSJNKRFB5G4YP7DFHDWOAYE23II3OA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.209662921 (58h26m7.080745966s) [noise=true , address=145.79.227.18:4010 , piece_id=S2EOTEV4TBNNDOBN7TIX76AJE2WI5P5DM3JK2NJB25TISYLS7VSQ, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.214493680 (58h26m7.085576725s) [noise=true , address=77.90.31.10:4539 , piece_id=27YFK55QLZDEULYABQ3OBDSPDDEIUJ2MJSD7X3A4K6HRFJMRQJUA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:08.555121654 (58h26m8.426204699s) [upload=51329]: Next returned nil segment Dec 8 01:55:08.555145519 (58h26m8.426228564s) [upload=51329]: Waiting for error group managing segments... Dec 8 01:55:08.935193756 (58h26m8.806276791s) [noise=true , address=64.49.13.27:4957 , piece_id=7METNUA4TAJ2FSYO5S4NXECVMMMCXKFL5CAHPQBQJ6WREQ3YFC3Q, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.093051314 (58h26m8.964134359s) [noise=true , address=145.79.227.16:4933 , piece_id=LHF64L3SWXFVZ5X5D67I44WSJNKRFB5G4YP7DFHDWOAYE23II3OA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.098812273 (58h26m8.969895318s) [noise=true , address=145.79.227.18:4010 , piece_id=S2EOTEV4TBNNDOBN7TIX76AJE2WI5P5DM3JK2NJB25TISYLS7VSQ, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.102616718 (58h26m8.973699763s) [noise=true , address=145.79.227.25:4627 , piece_id=K7UYU3W2CACUAQUUL72GG5F7DP5CHAHZ4M47TZG6NBN5Z6ZCDRBA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.118902949 (58h26m8.989985994s) [noise=true , address=145.79.227.34:4928 , piece_id=DRKF2DRSCBAYSZO66A2ZON2JS6HIJIT2YUWFZRS3W2CKFTV3NLVA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.124783838 (58h26m8.995866883s) [noise=true , address=77.90.31.18:4825 , piece_id=EJ4LO4TCEFB4OQQLFOEUYGQKK462ZPXXEIIY5CHF2PERXWDGQNDA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.126718282 (58h26m8.997801327s) [noise=true , address=145.79.227.11:4740 , piece_id=KL54LL2UQRT6MH2K3NQQJB3BKPX4QRWQMXRK27MW2U7CHFDCCWFA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.145158485 (58h26m9.01624153s) [noise=true , address=77.90.31.10:4550 , piece_id=PDCZ7VXNHQJ6LYPG64GZRYP4MRBGVPXCWTXSRT3UV2N6HGMDE2KA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.156650047 (58h26m9.027733092s) [noise=true , address=145.79.227.15:4908 , piece_id=JIUZSMOAKBFCFAX6RQ52TCNZMYBREQCYOEM75U3H2QYR4WH3KWCA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.162020363 (58h26m9.033103398s) [noise=true , address=77.90.31.10:4539 , piece_id=27YFK55QLZDEULYABQ3OBDSPDDEIUJ2MJSD7X3A4K6HRFJMRQJUA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.198899907 (58h26m9.069982942s) [noise=true , address=77.90.31.12:4621 , piece_id=S4FVJHEPIO2MVO56ZRIJ27RMTV723MERPLJLVD2T2GAO3T7K24TA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.208468337 (58h26m9.079551382s) [noise=true , address=45.91.116.4:4945 , piece_id=6UNWQQAWWWWC2XVUCDRCTAYJDTHBXAZJMXA6KLCTOP5PYSAJCCGA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.212217697 (58h26m9.083300732s) [noise=true , address=45.91.116.12:4254 , piece_id=UCKU4TBXJC4PSYHRDPUXETXLAILDOWA5NN7RCVVNYLVUBY7FBIFA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.215832390 (58h26m9.086915435s) [noise=true , address=77.90.31.11:4579 , piece_id=JINRNEVUAP2VI5Y4CR7ALWDQIAIAFX3SP2VVZSXDDG6XPUKASE3Q, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.215862086 (58h26m9.086945161s) [seg_pos={0 0}, upload=51329]: Segment reached optimal threshold of 14 pieces. Dec 8 01:55:09.216078098 (58h26m9.087161133s) [noise=true , address=45.91.116.6:4003 , piece_id=CRONTV2B2AVZTLKU3YNIEGQZSON7XHHL7S3K6KNNJV4YR5FHQ3PA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1veqoBGw6a8AGMoigyuNJMjcEvCT93xRq2B1d2KNhs6aEmSqVF): protocol: expected piece hash; context canceled; EOF Dec 8 01:55:09.216085662 (58h26m9.087168707s) [noise=true , address=45.91.116.6:4003 , piece_id=CRONTV2B2AVZTLKU3YNIEGQZSON7XHHL7S3K6KNNJV4YR5FHQ3PA, seg_pos={0 0}, upload=51329]: Long tail cancellation due to optimal threshold Dec 8 01:55:09.216307705 (58h26m9.08739075s) [noise=true , address=145.79.227.30:4786 , piece_id=VM65RUN3GUZZFUOHFYE7ZLPWU6COCQUMY3IK7GCNWPIUZAMEBR7Q, seg_pos={0 0}, upload=51329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125TWU7R4pkmrUxsNLhgTD475aLbjUJ7psA6drJtE7oDqZXYJ4Q): protocol: expected piece hash; context canceled; EOF Dec 8 01:55:09.216313956 (58h26m9.087396991s) [noise=true , address=145.79.227.30:4786 , piece_id=VM65RUN3GUZZFUOHFYE7ZLPWU6COCQUMY3IK7GCNWPIUZAMEBR7Q, seg_pos={0 0}, upload=51329]: Long tail cancellation due to optimal threshold Dec 8 01:55:09.216414809 (58h26m9.087497844s) [noise=true , address=77.90.31.16:4760 , piece_id=Q67LYTDJCAR2TFVSAIIYCUYM7D44JFPBLCSKRNLXH26EAW5UQ3GA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1e6p1TD5CqUyWdZjkrxh2kRqMuyDiMUr9eimfeayDAKwhtBrZ8): protocol: expected piece hash; context canceled; EOF Dec 8 01:55:09.216420681 (58h26m9.087503726s) [noise=true , address=77.90.31.16:4760 , piece_id=Q67LYTDJCAR2TFVSAIIYCUYM7D44JFPBLCSKRNLXH26EAW5UQ3GA, seg_pos={0 0}, upload=51329]: Long tail cancellation due to optimal threshold Dec 8 01:55:09.216562861 (58h26m9.087645906s) [noise=true , address=45.91.116.7:4037 , piece_id=UBNFDSXQBISC5QQSNGAAZIR7LRE7FCYRVYN5MMOY3A2ALEOQBYQQ, seg_pos={0 0}, upload=51329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iNPwE2XSfXcaXAXcrH2jcyNGYeDiiWvzAcDQffrmjPwTVqVmK): protocol: expected piece hash; context canceled; EOF Dec 8 01:55:09.216570025 (58h26m9.08765307s) [noise=true , address=45.91.116.7:4037 , piece_id=UBNFDSXQBISC5QQSNGAAZIR7LRE7FCYRVYN5MMOY3A2ALEOQBYQQ, seg_pos={0 0}, upload=51329]: Long tail cancellation due to optimal threshold Dec 8 01:55:09.216585564 (58h26m9.087668609s) [seg_pos={0 0}, upload=51329]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 8 01:55:09.216654365 (58h26m9.0877374s) [upload=51329]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 01:55:09.216659064 (58h26m9.087742099s) [upload=51329]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 01:55:35.852213564 (58h26m35.723296629s) [upload=51329]: Done uploading segments... Dec 8 01:55:35.852273619 (58h26m35.723356664s) [upload=51329]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.740295588s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 8.603132167s trace: https://jaeger.dev.storj.io/trace/27d92c196f473eec FAIL (2025-12-08 02:32:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 4891s) [upload=51469]: Got next segment. Inline: false Dec 8 02:04:21.143134547 (58h35m21.014217592s) [upload=51469]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 02:04:21.143141259 (58h35m21.014224304s) [upload=51469]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 02:04:22.267502706 (58h35m22.138585751s) [seg_pos={0 0}, upload=51469]: Begin upload segment... Dec 8 02:04:22.267624127 (58h35m22.138707172s) [seg_pos={0 0}, upload=51469]: Done begin upload segment. Dec 8 02:04:22.267698027 (58h35m22.138781073s) [noise=true , address=77.90.31.11:4554 , piece_id=BQP7XTRA2SBVINT4J7JAOF3ORNENBBIQN4ABMCEMQQ6FQL6QUOKA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.267712997 (58h35m22.138796042s) [noise=true , address=145.79.227.41:4087 , piece_id=XO6K3CXK7KZ5VEVXULHKQXLSSLIIWIX3VKQOSD7HY25WMNMCDLEQ, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.268434680 (58h35m22.139517715s) [noise=true , address=145.79.227.16:4910 , piece_id=ETE7ZBHT5IJUOBWCA5BGLKC2ANDEUB2M3DOMXWWS2RBWSJQBARRQ, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.268550331 (58h35m22.139633366s) [noise=true , address=145.79.227.26:4653 , piece_id=DKW72PEWPIOHPYHLI54PBU7NR3HGUBLFM3MN4XQQFWIIZ4FHIUNA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.268913111 (58h35m22.139996146s) [noise=true , address=145.79.227.23:4545 , piece_id=NBYCSU2KSHTAL223AHKHREQWKXUEQKJU46TY3SPYTKT5N5T2OUSA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.268924032 (58h35m22.140007067s) [noise=true , address=145.79.227.40:4051 , piece_id=7CQTGHFZ542WUHYX2CTOPN4PF2KDOOLDGV64THDBZCYOCXA3JWDA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.269398886 (58h35m22.140481921s) [noise=true , address=77.90.31.10:4518 , piece_id=6AR3NAFODP54SKG3SJSJSIQQFOYXO2C65WVNN4Z4XG426ZEOEDQA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.269414686 (58h35m22.140497721s) [noise=true , address=145.79.227.25:4619 , piece_id=XBNGVRIQLQUWKKBF4ULL7ZJDDT75QLY7ES2J7WHYWRD2ZLHOGU2Q, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.269753261 (58h35m22.140836296s) [noise=true , address=145.79.227.23:4547 , piece_id=5Y6V5QWLCWNOLT2GDNUEIRSG75KGNGIFLBCBGSIT7XH46RZYVDGQ, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.270016322 (58h35m22.141099367s) [noise=true , address=77.90.31.13:4628 , piece_id=LWTOCQRRA7HLI5ZPPRQ25IM22LMQNRR5H3TTOTSTGCFOYFZMABAA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.270259175 (58h35m22.14134222s) [noise=true , address=145.79.227.25:4617 , piece_id=S4ETCJ6WMCPXXQCYPPBBMHLMK2SVJV4S5VOKHGTDAGSGG6WRYH3Q, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.270655339 (58h35m22.141738374s) [noise=true , address=145.79.227.38:4977 , piece_id=73WXA33TWM4MFVTHDHW3PXJVABMCVMQYI3P42AHT3OYPKMXIVTUQ, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.271143158 (58h35m22.142246412s) [noise=true , address=145.79.227.28:4727 , piece_id=UYRK457JBF2AHCIXZ4XOIL7YHOVYZ7IPDUTB6WOZZUDEW5Q7NNCQ, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.271230565 (58h35m22.14231361s) [noise=true , address=145.79.227.39:4013 , piece_id=UI22RDXJ2DY3RFNSJVNT43XQJ6B2S3EO6SFLYHXW4VPVQJSYYF6Q, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.271662086 (58h35m22.142745121s) [noise=true , address=145.79.227.22:4511 , piece_id=GGA463DAOSZCW7SQGZW2LGFRQVNUOUJ2L2GXRXLSDPC65KVPY3TA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.272592658 (58h35m22.143675703s) [noise=true , address=145.79.227.24:4583 , piece_id=WYTXGXG77ZW4RGWX7YWDU2NH6R5CCPVBPQH3ISHUAJVKZJRONS5Q, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:24.009199123 (58h35m23.880282158s) [upload=51469]: Next returned nil segment Dec 8 02:04:24.009221235 (58h35m23.88030428s) [upload=51469]: Waiting for error group managing segments... Dec 8 02:04:24.293906896 (58h35m24.164989931s) [noise=true , address=145.79.227.24:4583 , piece_id=WYTXGXG77ZW4RGWX7YWDU2NH6R5CCPVBPQH3ISHUAJVKZJRONS5Q, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.298576738 (58h35m24.169659783s) [noise=true , address=145.79.227.28:4727 , piece_id=UYRK457JBF2AHCIXZ4XOIL7YHOVYZ7IPDUTB6WOZZUDEW5Q7NNCQ, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.301356571 (58h35m24.172439616s) [noise=true , address=145.79.227.25:4619 , piece_id=XBNGVRIQLQUWKKBF4ULL7ZJDDT75QLY7ES2J7WHYWRD2ZLHOGU2Q, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.326466765 (58h35m24.19754981s) [noise=true , address=145.79.227.16:4910 , piece_id=ETE7ZBHT5IJUOBWCA5BGLKC2ANDEUB2M3DOMXWWS2RBWSJQBARRQ, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.328633530 (58h35m24.199716586s) [noise=true , address=145.79.227.26:4653 , piece_id=DKW72PEWPIOHPYHLI54PBU7NR3HGUBLFM3MN4XQQFWIIZ4FHIUNA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.360508106 (58h35m24.231591151s) [noise=true , address=145.79.227.40:4051 , piece_id=7CQTGHFZ542WUHYX2CTOPN4PF2KDOOLDGV64THDBZCYOCXA3JWDA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.361116263 (58h35m24.232199308s) [noise=true , address=145.79.227.25:4617 , piece_id=S4ETCJ6WMCPXXQCYPPBBMHLMK2SVJV4S5VOKHGTDAGSGG6WRYH3Q, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.361695086 (58h35m24.232778131s) [noise=true , address=145.79.227.41:4087 , piece_id=XO6K3CXK7KZ5VEVXULHKQXLSSLIIWIX3VKQOSD7HY25WMNMCDLEQ, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.371520525 (58h35m24.24260356s) [noise=true , address=145.79.227.38:4977 , piece_id=73WXA33TWM4MFVTHDHW3PXJVABMCVMQYI3P42AHT3OYPKMXIVTUQ, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.381271893 (58h35m24.252354928s) [noise=true , address=145.79.227.23:4545 , piece_id=NBYCSU2KSHTAL223AHKHREQWKXUEQKJU46TY3SPYTKT5N5T2OUSA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.386403053 (58h35m24.257486098s) [noise=true , address=77.90.31.11:4554 , piece_id=BQP7XTRA2SBVINT4J7JAOF3ORNENBBIQN4ABMCEMQQ6FQL6QUOKA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.386427209 (58h35m24.257510254s) [noise=true , address=77.90.31.10:4518 , piece_id=6AR3NAFODP54SKG3SJSJSIQQFOYXO2C65WVNN4Z4XG426ZEOEDQA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.400489386 (58h35m24.271572431s) [noise=true , address=77.90.31.13:4628 , piece_id=LWTOCQRRA7HLI5ZPPRQ25IM22LMQNRR5H3TTOTSTGCFOYFZMABAA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.400513903 (58h35m24.271596938s) [seg_pos={0 0}, upload=51469]: Segment reached optimal threshold of 13 pieces. Dec 8 02:04:24.400813233 (58h35m24.271896288s) [noise=true , address=145.79.227.39:4013 , piece_id=UI22RDXJ2DY3RFNSJVNT43XQJ6B2S3EO6SFLYHXW4VPVQJSYYF6Q, seg_pos={0 0}, upload=51469]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b): protocol: write tcp 188.166.225.170:39434->145.79.227.39:4013: use of closed network connection; write tcp 188.166.225.170:39434->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39434->145.79.227.39:4013: use of closed network connection Dec 8 02:04:24.400825677 (58h35m24.271908722s) [noise=true , address=145.79.227.39:4013 , piece_id=UI22RDXJ2DY3RFNSJVNT43XQJ6B2S3EO6SFLYHXW4VPVQJSYYF6Q, seg_pos={0 0}, upload=51469]: Long tail cancellation due to optimal threshold Dec 8 02:04:24.400873027 (58h35m24.271956082s) [noise=true , address=145.79.227.22:4511 , piece_id=GGA463DAOSZCW7SQGZW2LGFRQVNUOUJ2L2GXRXLSDPC65KVPY3TA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG): protocol: write tcp 188.166.225.170:47926->145.79.227.22:4511: use of closed network connection; write tcp 188.166.225.170:47926->145.79.227.22:4511: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47926->145.79.227.22:4511: use of closed network connection Dec 8 02:04:24.400894248 (58h35m24.271977293s) [noise=true , address=145.79.227.22:4511 , piece_id=GGA463DAOSZCW7SQGZW2LGFRQVNUOUJ2L2GXRXLSDPC65KVPY3TA, seg_pos={0 0}, upload=51469]: Long tail cancellation due to optimal threshold Dec 8 02:04:24.401247129 (58h35m24.272330184s) [noise=true , address=145.79.227.23:4547 , piece_id=5Y6V5QWLCWNOLT2GDNUEIRSG75KGNGIFLBCBGSIT7XH46RZYVDGQ, seg_pos={0 0}, upload=51469]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW): protocol: read tcp 188.166.225.170:43344->145.79.227.23:4547: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 02:04:24.401260003 (58h35m24.272343059s) [noise=true , address=145.79.227.23:4547 , piece_id=5Y6V5QWLCWNOLT2GDNUEIRSG75KGNGIFLBCBGSIT7XH46RZYVDGQ, seg_pos={0 0}, upload=51469]: Long tail cancellation due to optimal threshold Dec 8 02:04:24.401285452 (58h35m24.272368497s) [seg_pos={0 0}, upload=51469]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 02:04:24.401382247 (58h35m24.272465292s) [upload=51469]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 02:04:24.401391394 (58h35m24.272474429s) [upload=51469]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 02:05:39.049922629 (58h36m38.921005704s) [upload=51469]: Done uploading segments... Dec 8 02:05:39.050003342 (58h36m38.921086377s) [upload=51469]: Upload finished. err: metaclient: context deadline exceeded Duration: 8.603158527s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [..............X......X................X....X................] duration: 3.528483995s trace: https://jaeger.dev.storj.io/trace/491130a80ad889c4 FAIL (2025-12-08 02:31:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.528537939s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 5.672812859s trace: https://jaeger.dev.storj.io/trace/4030003299248e86 FAIL (2025-12-08 02:31:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.672915456s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 5.681006352s trace: https://jaeger.dev.storj.io/trace/49d209695fd399e8 FAIL (2025-12-08 02:31:52 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 5.681046678s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........X.........X.......................................] duration: 7.066420862s trace: https://jaeger.dev.storj.io/trace/e7164007ac0fd55 FAIL (2025-12-08 02:31:44 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 7.066448173s core uplink us1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 4.859064642s trace: https://jaeger.dev.storj.io/trace/7af096edea7ffcf5 FAIL (2025-12-08 02:31:38 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.859090692s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 5.856328496s trace: https://jaeger.dev.storj.io/trace/4030003299248e86 FAIL (2025-12-08 02:31:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.856355507s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 8.070844448s trace: https://jaeger.dev.storj.io/trace/27d92c196f473eec FAIL (2025-12-08 02:31:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 4891s) [upload=51469]: Got next segment. Inline: false Dec 8 02:04:21.143134547 (58h35m21.014217592s) [upload=51469]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 02:04:21.143141259 (58h35m21.014224304s) [upload=51469]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 02:04:22.267502706 (58h35m22.138585751s) [seg_pos={0 0}, upload=51469]: Begin upload segment... Dec 8 02:04:22.267624127 (58h35m22.138707172s) [seg_pos={0 0}, upload=51469]: Done begin upload segment. Dec 8 02:04:22.267698027 (58h35m22.138781073s) [noise=true , address=77.90.31.11:4554 , piece_id=BQP7XTRA2SBVINT4J7JAOF3ORNENBBIQN4ABMCEMQQ6FQL6QUOKA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.267712997 (58h35m22.138796042s) [noise=true , address=145.79.227.41:4087 , piece_id=XO6K3CXK7KZ5VEVXULHKQXLSSLIIWIX3VKQOSD7HY25WMNMCDLEQ, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.268434680 (58h35m22.139517715s) [noise=true , address=145.79.227.16:4910 , piece_id=ETE7ZBHT5IJUOBWCA5BGLKC2ANDEUB2M3DOMXWWS2RBWSJQBARRQ, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.268550331 (58h35m22.139633366s) [noise=true , address=145.79.227.26:4653 , piece_id=DKW72PEWPIOHPYHLI54PBU7NR3HGUBLFM3MN4XQQFWIIZ4FHIUNA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.268913111 (58h35m22.139996146s) [noise=true , address=145.79.227.23:4545 , piece_id=NBYCSU2KSHTAL223AHKHREQWKXUEQKJU46TY3SPYTKT5N5T2OUSA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.268924032 (58h35m22.140007067s) [noise=true , address=145.79.227.40:4051 , piece_id=7CQTGHFZ542WUHYX2CTOPN4PF2KDOOLDGV64THDBZCYOCXA3JWDA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.269398886 (58h35m22.140481921s) [noise=true , address=77.90.31.10:4518 , piece_id=6AR3NAFODP54SKG3SJSJSIQQFOYXO2C65WVNN4Z4XG426ZEOEDQA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.269414686 (58h35m22.140497721s) [noise=true , address=145.79.227.25:4619 , piece_id=XBNGVRIQLQUWKKBF4ULL7ZJDDT75QLY7ES2J7WHYWRD2ZLHOGU2Q, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.269753261 (58h35m22.140836296s) [noise=true , address=145.79.227.23:4547 , piece_id=5Y6V5QWLCWNOLT2GDNUEIRSG75KGNGIFLBCBGSIT7XH46RZYVDGQ, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.270016322 (58h35m22.141099367s) [noise=true , address=77.90.31.13:4628 , piece_id=LWTOCQRRA7HLI5ZPPRQ25IM22LMQNRR5H3TTOTSTGCFOYFZMABAA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.270259175 (58h35m22.14134222s) [noise=true , address=145.79.227.25:4617 , piece_id=S4ETCJ6WMCPXXQCYPPBBMHLMK2SVJV4S5VOKHGTDAGSGG6WRYH3Q, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.270655339 (58h35m22.141738374s) [noise=true , address=145.79.227.38:4977 , piece_id=73WXA33TWM4MFVTHDHW3PXJVABMCVMQYI3P42AHT3OYPKMXIVTUQ, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.271143158 (58h35m22.142246412s) [noise=true , address=145.79.227.28:4727 , piece_id=UYRK457JBF2AHCIXZ4XOIL7YHOVYZ7IPDUTB6WOZZUDEW5Q7NNCQ, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.271230565 (58h35m22.14231361s) [noise=true , address=145.79.227.39:4013 , piece_id=UI22RDXJ2DY3RFNSJVNT43XQJ6B2S3EO6SFLYHXW4VPVQJSYYF6Q, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.271662086 (58h35m22.142745121s) [noise=true , address=145.79.227.22:4511 , piece_id=GGA463DAOSZCW7SQGZW2LGFRQVNUOUJ2L2GXRXLSDPC65KVPY3TA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.272592658 (58h35m22.143675703s) [noise=true , address=145.79.227.24:4583 , piece_id=WYTXGXG77ZW4RGWX7YWDU2NH6R5CCPVBPQH3ISHUAJVKZJRONS5Q, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:24.009199123 (58h35m23.880282158s) [upload=51469]: Next returned nil segment Dec 8 02:04:24.009221235 (58h35m23.88030428s) [upload=51469]: Waiting for error group managing segments... Dec 8 02:04:24.293906896 (58h35m24.164989931s) [noise=true , address=145.79.227.24:4583 , piece_id=WYTXGXG77ZW4RGWX7YWDU2NH6R5CCPVBPQH3ISHUAJVKZJRONS5Q, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.298576738 (58h35m24.169659783s) [noise=true , address=145.79.227.28:4727 , piece_id=UYRK457JBF2AHCIXZ4XOIL7YHOVYZ7IPDUTB6WOZZUDEW5Q7NNCQ, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.301356571 (58h35m24.172439616s) [noise=true , address=145.79.227.25:4619 , piece_id=XBNGVRIQLQUWKKBF4ULL7ZJDDT75QLY7ES2J7WHYWRD2ZLHOGU2Q, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.326466765 (58h35m24.19754981s) [noise=true , address=145.79.227.16:4910 , piece_id=ETE7ZBHT5IJUOBWCA5BGLKC2ANDEUB2M3DOMXWWS2RBWSJQBARRQ, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.328633530 (58h35m24.199716586s) [noise=true , address=145.79.227.26:4653 , piece_id=DKW72PEWPIOHPYHLI54PBU7NR3HGUBLFM3MN4XQQFWIIZ4FHIUNA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.360508106 (58h35m24.231591151s) [noise=true , address=145.79.227.40:4051 , piece_id=7CQTGHFZ542WUHYX2CTOPN4PF2KDOOLDGV64THDBZCYOCXA3JWDA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.361116263 (58h35m24.232199308s) [noise=true , address=145.79.227.25:4617 , piece_id=S4ETCJ6WMCPXXQCYPPBBMHLMK2SVJV4S5VOKHGTDAGSGG6WRYH3Q, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.361695086 (58h35m24.232778131s) [noise=true , address=145.79.227.41:4087 , piece_id=XO6K3CXK7KZ5VEVXULHKQXLSSLIIWIX3VKQOSD7HY25WMNMCDLEQ, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.371520525 (58h35m24.24260356s) [noise=true , address=145.79.227.38:4977 , piece_id=73WXA33TWM4MFVTHDHW3PXJVABMCVMQYI3P42AHT3OYPKMXIVTUQ, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.381271893 (58h35m24.252354928s) [noise=true , address=145.79.227.23:4545 , piece_id=NBYCSU2KSHTAL223AHKHREQWKXUEQKJU46TY3SPYTKT5N5T2OUSA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.386403053 (58h35m24.257486098s) [noise=true , address=77.90.31.11:4554 , piece_id=BQP7XTRA2SBVINT4J7JAOF3ORNENBBIQN4ABMCEMQQ6FQL6QUOKA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.386427209 (58h35m24.257510254s) [noise=true , address=77.90.31.10:4518 , piece_id=6AR3NAFODP54SKG3SJSJSIQQFOYXO2C65WVNN4Z4XG426ZEOEDQA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.400489386 (58h35m24.271572431s) [noise=true , address=77.90.31.13:4628 , piece_id=LWTOCQRRA7HLI5ZPPRQ25IM22LMQNRR5H3TTOTSTGCFOYFZMABAA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.400513903 (58h35m24.271596938s) [seg_pos={0 0}, upload=51469]: Segment reached optimal threshold of 13 pieces. Dec 8 02:04:24.400813233 (58h35m24.271896288s) [noise=true , address=145.79.227.39:4013 , piece_id=UI22RDXJ2DY3RFNSJVNT43XQJ6B2S3EO6SFLYHXW4VPVQJSYYF6Q, seg_pos={0 0}, upload=51469]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b): protocol: write tcp 188.166.225.170:39434->145.79.227.39:4013: use of closed network connection; write tcp 188.166.225.170:39434->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39434->145.79.227.39:4013: use of closed network connection Dec 8 02:04:24.400825677 (58h35m24.271908722s) [noise=true , address=145.79.227.39:4013 , piece_id=UI22RDXJ2DY3RFNSJVNT43XQJ6B2S3EO6SFLYHXW4VPVQJSYYF6Q, seg_pos={0 0}, upload=51469]: Long tail cancellation due to optimal threshold Dec 8 02:04:24.400873027 (58h35m24.271956082s) [noise=true , address=145.79.227.22:4511 , piece_id=GGA463DAOSZCW7SQGZW2LGFRQVNUOUJ2L2GXRXLSDPC65KVPY3TA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG): protocol: write tcp 188.166.225.170:47926->145.79.227.22:4511: use of closed network connection; write tcp 188.166.225.170:47926->145.79.227.22:4511: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47926->145.79.227.22:4511: use of closed network connection Dec 8 02:04:24.400894248 (58h35m24.271977293s) [noise=true , address=145.79.227.22:4511 , piece_id=GGA463DAOSZCW7SQGZW2LGFRQVNUOUJ2L2GXRXLSDPC65KVPY3TA, seg_pos={0 0}, upload=51469]: Long tail cancellation due to optimal threshold Dec 8 02:04:24.401247129 (58h35m24.272330184s) [noise=true , address=145.79.227.23:4547 , piece_id=5Y6V5QWLCWNOLT2GDNUEIRSG75KGNGIFLBCBGSIT7XH46RZYVDGQ, seg_pos={0 0}, upload=51469]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW): protocol: read tcp 188.166.225.170:43344->145.79.227.23:4547: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 02:04:24.401260003 (58h35m24.272343059s) [noise=true , address=145.79.227.23:4547 , piece_id=5Y6V5QWLCWNOLT2GDNUEIRSG75KGNGIFLBCBGSIT7XH46RZYVDGQ, seg_pos={0 0}, upload=51469]: Long tail cancellation due to optimal threshold Dec 8 02:04:24.401285452 (58h35m24.272368497s) [seg_pos={0 0}, upload=51469]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 02:04:24.401382247 (58h35m24.272465292s) [upload=51469]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 02:04:24.401391394 (58h35m24.272474429s) [upload=51469]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 02:05:39.049922629 (58h36m38.921005704s) [upload=51469]: Done uploading segments... Dec 8 02:05:39.050003342 (58h36m38.921086377s) [upload=51469]: Upload finished. err: metaclient: context deadline exceeded Duration: 8.070870557s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 6.717994377s trace: https://jaeger.dev.storj.io/trace/57258282a8e974fd FAIL (2025-12-08 02:31:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== UA4TAJ2FSYO5S4NXECVMMMCXKFL5CAHPQBQJ6WREQ3YFC3Q, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.202906305 (58h26m7.07398934s) [noise=true , address=45.91.116.4:4945 , piece_id=6UNWQQAWWWWC2XVUCDRCTAYJDTHBXAZJMXA6KLCTOP5PYSAJCCGA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.203371201 (58h26m7.074454246s) [noise=true , address=145.79.227.11:4740 , piece_id=KL54LL2UQRT6MH2K3NQQJB3BKPX4QRWQMXRK27MW2U7CHFDCCWFA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.203861354 (58h26m7.07494439s) [noise=true , address=77.90.31.12:4621 , piece_id=S4FVJHEPIO2MVO56ZRIJ27RMTV723MERPLJLVD2T2GAO3T7K24TA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.204290401 (58h26m7.075373446s) [noise=true , address=145.79.227.34:4928 , piece_id=DRKF2DRSCBAYSZO66A2ZON2JS6HIJIT2YUWFZRS3W2CKFTV3NLVA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.204711363 (58h26m7.075794398s) [noise=true , address=145.79.227.15:4908 , piece_id=JIUZSMOAKBFCFAX6RQ52TCNZMYBREQCYOEM75U3H2QYR4WH3KWCA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.205118208 (58h26m7.076201253s) [noise=true , address=77.90.31.18:4825 , piece_id=EJ4LO4TCEFB4OQQLFOEUYGQKK462ZPXXEIIY5CHF2PERXWDGQNDA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.205554277 (58h26m7.076637323s) [noise=true , address=145.79.227.30:4786 , piece_id=VM65RUN3GUZZFUOHFYE7ZLPWU6COCQUMY3IK7GCNWPIUZAMEBR7Q, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.206116137 (58h26m7.077199182s) [noise=true , address=145.79.227.25:4627 , piece_id=K7UYU3W2CACUAQUUL72GG5F7DP5CHAHZ4M47TZG6NBN5Z6ZCDRBA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.206664562 (58h26m7.077747607s) [noise=true , address=77.90.31.16:4760 , piece_id=Q67LYTDJCAR2TFVSAIIYCUYM7D44JFPBLCSKRNLXH26EAW5UQ3GA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.207102556 (58h26m7.078185601s) [noise=true , address=45.91.116.7:4037 , piece_id=UBNFDSXQBISC5QQSNGAAZIR7LRE7FCYRVYN5MMOY3A2ALEOQBYQQ, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.207519911 (58h26m7.078602956s) [noise=true , address=77.90.31.11:4579 , piece_id=JINRNEVUAP2VI5Y4CR7ALWDQIAIAFX3SP2VVZSXDDG6XPUKASE3Q, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.207967393 (58h26m7.079050438s) [noise=true , address=77.90.31.10:4550 , piece_id=PDCZ7VXNHQJ6LYPG64GZRYP4MRBGVPXCWTXSRT3UV2N6HGMDE2KA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.208416427 (58h26m7.079499472s) [noise=true , address=45.91.116.6:4003 , piece_id=CRONTV2B2AVZTLKU3YNIEGQZSON7XHHL7S3K6KNNJV4YR5FHQ3PA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.208830746 (58h26m7.079913791s) [noise=true , address=45.91.116.12:4254 , piece_id=UCKU4TBXJC4PSYHRDPUXETXLAILDOWA5NN7RCVVNYLVUBY7FBIFA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.209255245 (58h26m7.080338291s) [noise=true , address=145.79.227.16:4933 , piece_id=LHF64L3SWXFVZ5X5D67I44WSJNKRFB5G4YP7DFHDWOAYE23II3OA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.209662921 (58h26m7.080745966s) [noise=true , address=145.79.227.18:4010 , piece_id=S2EOTEV4TBNNDOBN7TIX76AJE2WI5P5DM3JK2NJB25TISYLS7VSQ, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.214493680 (58h26m7.085576725s) [noise=true , address=77.90.31.10:4539 , piece_id=27YFK55QLZDEULYABQ3OBDSPDDEIUJ2MJSD7X3A4K6HRFJMRQJUA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:08.555121654 (58h26m8.426204699s) [upload=51329]: Next returned nil segment Dec 8 01:55:08.555145519 (58h26m8.426228564s) [upload=51329]: Waiting for error group managing segments... Dec 8 01:55:08.935193756 (58h26m8.806276791s) [noise=true , address=64.49.13.27:4957 , piece_id=7METNUA4TAJ2FSYO5S4NXECVMMMCXKFL5CAHPQBQJ6WREQ3YFC3Q, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.093051314 (58h26m8.964134359s) [noise=true , address=145.79.227.16:4933 , piece_id=LHF64L3SWXFVZ5X5D67I44WSJNKRFB5G4YP7DFHDWOAYE23II3OA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.098812273 (58h26m8.969895318s) [noise=true , address=145.79.227.18:4010 , piece_id=S2EOTEV4TBNNDOBN7TIX76AJE2WI5P5DM3JK2NJB25TISYLS7VSQ, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.102616718 (58h26m8.973699763s) [noise=true , address=145.79.227.25:4627 , piece_id=K7UYU3W2CACUAQUUL72GG5F7DP5CHAHZ4M47TZG6NBN5Z6ZCDRBA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.118902949 (58h26m8.989985994s) [noise=true , address=145.79.227.34:4928 , piece_id=DRKF2DRSCBAYSZO66A2ZON2JS6HIJIT2YUWFZRS3W2CKFTV3NLVA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.124783838 (58h26m8.995866883s) [noise=true , address=77.90.31.18:4825 , piece_id=EJ4LO4TCEFB4OQQLFOEUYGQKK462ZPXXEIIY5CHF2PERXWDGQNDA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.126718282 (58h26m8.997801327s) [noise=true , address=145.79.227.11:4740 , piece_id=KL54LL2UQRT6MH2K3NQQJB3BKPX4QRWQMXRK27MW2U7CHFDCCWFA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.145158485 (58h26m9.01624153s) [noise=true , address=77.90.31.10:4550 , piece_id=PDCZ7VXNHQJ6LYPG64GZRYP4MRBGVPXCWTXSRT3UV2N6HGMDE2KA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.156650047 (58h26m9.027733092s) [noise=true , address=145.79.227.15:4908 , piece_id=JIUZSMOAKBFCFAX6RQ52TCNZMYBREQCYOEM75U3H2QYR4WH3KWCA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.162020363 (58h26m9.033103398s) [noise=true , address=77.90.31.10:4539 , piece_id=27YFK55QLZDEULYABQ3OBDSPDDEIUJ2MJSD7X3A4K6HRFJMRQJUA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.198899907 (58h26m9.069982942s) [noise=true , address=77.90.31.12:4621 , piece_id=S4FVJHEPIO2MVO56ZRIJ27RMTV723MERPLJLVD2T2GAO3T7K24TA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.208468337 (58h26m9.079551382s) [noise=true , address=45.91.116.4:4945 , piece_id=6UNWQQAWWWWC2XVUCDRCTAYJDTHBXAZJMXA6KLCTOP5PYSAJCCGA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.212217697 (58h26m9.083300732s) [noise=true , address=45.91.116.12:4254 , piece_id=UCKU4TBXJC4PSYHRDPUXETXLAILDOWA5NN7RCVVNYLVUBY7FBIFA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.215832390 (58h26m9.086915435s) [noise=true , address=77.90.31.11:4579 , piece_id=JINRNEVUAP2VI5Y4CR7ALWDQIAIAFX3SP2VVZSXDDG6XPUKASE3Q, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.215862086 (58h26m9.086945161s) [seg_pos={0 0}, upload=51329]: Segment reached optimal threshold of 14 pieces. Dec 8 01:55:09.216078098 (58h26m9.087161133s) [noise=true , address=45.91.116.6:4003 , piece_id=CRONTV2B2AVZTLKU3YNIEGQZSON7XHHL7S3K6KNNJV4YR5FHQ3PA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1veqoBGw6a8AGMoigyuNJMjcEvCT93xRq2B1d2KNhs6aEmSqVF): protocol: expected piece hash; context canceled; EOF Dec 8 01:55:09.216085662 (58h26m9.087168707s) [noise=true , address=45.91.116.6:4003 , piece_id=CRONTV2B2AVZTLKU3YNIEGQZSON7XHHL7S3K6KNNJV4YR5FHQ3PA, seg_pos={0 0}, upload=51329]: Long tail cancellation due to optimal threshold Dec 8 01:55:09.216307705 (58h26m9.08739075s) [noise=true , address=145.79.227.30:4786 , piece_id=VM65RUN3GUZZFUOHFYE7ZLPWU6COCQUMY3IK7GCNWPIUZAMEBR7Q, seg_pos={0 0}, upload=51329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125TWU7R4pkmrUxsNLhgTD475aLbjUJ7psA6drJtE7oDqZXYJ4Q): protocol: expected piece hash; context canceled; EOF Dec 8 01:55:09.216313956 (58h26m9.087396991s) [noise=true , address=145.79.227.30:4786 , piece_id=VM65RUN3GUZZFUOHFYE7ZLPWU6COCQUMY3IK7GCNWPIUZAMEBR7Q, seg_pos={0 0}, upload=51329]: Long tail cancellation due to optimal threshold Dec 8 01:55:09.216414809 (58h26m9.087497844s) [noise=true , address=77.90.31.16:4760 , piece_id=Q67LYTDJCAR2TFVSAIIYCUYM7D44JFPBLCSKRNLXH26EAW5UQ3GA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1e6p1TD5CqUyWdZjkrxh2kRqMuyDiMUr9eimfeayDAKwhtBrZ8): protocol: expected piece hash; context canceled; EOF Dec 8 01:55:09.216420681 (58h26m9.087503726s) [noise=true , address=77.90.31.16:4760 , piece_id=Q67LYTDJCAR2TFVSAIIYCUYM7D44JFPBLCSKRNLXH26EAW5UQ3GA, seg_pos={0 0}, upload=51329]: Long tail cancellation due to optimal threshold Dec 8 01:55:09.216562861 (58h26m9.087645906s) [noise=true , address=45.91.116.7:4037 , piece_id=UBNFDSXQBISC5QQSNGAAZIR7LRE7FCYRVYN5MMOY3A2ALEOQBYQQ, seg_pos={0 0}, upload=51329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iNPwE2XSfXcaXAXcrH2jcyNGYeDiiWvzAcDQffrmjPwTVqVmK): protocol: expected piece hash; context canceled; EOF Dec 8 01:55:09.216570025 (58h26m9.08765307s) [noise=true , address=45.91.116.7:4037 , piece_id=UBNFDSXQBISC5QQSNGAAZIR7LRE7FCYRVYN5MMOY3A2ALEOQBYQQ, seg_pos={0 0}, upload=51329]: Long tail cancellation due to optimal threshold Dec 8 01:55:09.216585564 (58h26m9.087668609s) [seg_pos={0 0}, upload=51329]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 8 01:55:09.216654365 (58h26m9.0877374s) [upload=51329]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 01:55:09.216659064 (58h26m9.087742099s) [upload=51329]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 01:55:35.852213564 (58h26m35.723296629s) [upload=51329]: Done uploading segments... Dec 8 01:55:35.852273619 (58h26m35.723356664s) [upload=51329]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.718024795s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [...............X......X................X....X...............] duration: 3.685726392s trace: https://jaeger.dev.storj.io/trace/491130a80ad889c4 FAIL (2025-12-08 02:31:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.685763974s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 6.307198441s trace: https://jaeger.dev.storj.io/trace/57258282a8e974fd FAIL (2025-12-08 02:30:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== UA4TAJ2FSYO5S4NXECVMMMCXKFL5CAHPQBQJ6WREQ3YFC3Q, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.202906305 (58h26m7.07398934s) [noise=true , address=45.91.116.4:4945 , piece_id=6UNWQQAWWWWC2XVUCDRCTAYJDTHBXAZJMXA6KLCTOP5PYSAJCCGA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.203371201 (58h26m7.074454246s) [noise=true , address=145.79.227.11:4740 , piece_id=KL54LL2UQRT6MH2K3NQQJB3BKPX4QRWQMXRK27MW2U7CHFDCCWFA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.203861354 (58h26m7.07494439s) [noise=true , address=77.90.31.12:4621 , piece_id=S4FVJHEPIO2MVO56ZRIJ27RMTV723MERPLJLVD2T2GAO3T7K24TA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.204290401 (58h26m7.075373446s) [noise=true , address=145.79.227.34:4928 , piece_id=DRKF2DRSCBAYSZO66A2ZON2JS6HIJIT2YUWFZRS3W2CKFTV3NLVA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.204711363 (58h26m7.075794398s) [noise=true , address=145.79.227.15:4908 , piece_id=JIUZSMOAKBFCFAX6RQ52TCNZMYBREQCYOEM75U3H2QYR4WH3KWCA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.205118208 (58h26m7.076201253s) [noise=true , address=77.90.31.18:4825 , piece_id=EJ4LO4TCEFB4OQQLFOEUYGQKK462ZPXXEIIY5CHF2PERXWDGQNDA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.205554277 (58h26m7.076637323s) [noise=true , address=145.79.227.30:4786 , piece_id=VM65RUN3GUZZFUOHFYE7ZLPWU6COCQUMY3IK7GCNWPIUZAMEBR7Q, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.206116137 (58h26m7.077199182s) [noise=true , address=145.79.227.25:4627 , piece_id=K7UYU3W2CACUAQUUL72GG5F7DP5CHAHZ4M47TZG6NBN5Z6ZCDRBA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.206664562 (58h26m7.077747607s) [noise=true , address=77.90.31.16:4760 , piece_id=Q67LYTDJCAR2TFVSAIIYCUYM7D44JFPBLCSKRNLXH26EAW5UQ3GA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.207102556 (58h26m7.078185601s) [noise=true , address=45.91.116.7:4037 , piece_id=UBNFDSXQBISC5QQSNGAAZIR7LRE7FCYRVYN5MMOY3A2ALEOQBYQQ, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.207519911 (58h26m7.078602956s) [noise=true , address=77.90.31.11:4579 , piece_id=JINRNEVUAP2VI5Y4CR7ALWDQIAIAFX3SP2VVZSXDDG6XPUKASE3Q, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.207967393 (58h26m7.079050438s) [noise=true , address=77.90.31.10:4550 , piece_id=PDCZ7VXNHQJ6LYPG64GZRYP4MRBGVPXCWTXSRT3UV2N6HGMDE2KA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.208416427 (58h26m7.079499472s) [noise=true , address=45.91.116.6:4003 , piece_id=CRONTV2B2AVZTLKU3YNIEGQZSON7XHHL7S3K6KNNJV4YR5FHQ3PA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.208830746 (58h26m7.079913791s) [noise=true , address=45.91.116.12:4254 , piece_id=UCKU4TBXJC4PSYHRDPUXETXLAILDOWA5NN7RCVVNYLVUBY7FBIFA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.209255245 (58h26m7.080338291s) [noise=true , address=145.79.227.16:4933 , piece_id=LHF64L3SWXFVZ5X5D67I44WSJNKRFB5G4YP7DFHDWOAYE23II3OA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.209662921 (58h26m7.080745966s) [noise=true , address=145.79.227.18:4010 , piece_id=S2EOTEV4TBNNDOBN7TIX76AJE2WI5P5DM3JK2NJB25TISYLS7VSQ, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.214493680 (58h26m7.085576725s) [noise=true , address=77.90.31.10:4539 , piece_id=27YFK55QLZDEULYABQ3OBDSPDDEIUJ2MJSD7X3A4K6HRFJMRQJUA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:08.555121654 (58h26m8.426204699s) [upload=51329]: Next returned nil segment Dec 8 01:55:08.555145519 (58h26m8.426228564s) [upload=51329]: Waiting for error group managing segments... Dec 8 01:55:08.935193756 (58h26m8.806276791s) [noise=true , address=64.49.13.27:4957 , piece_id=7METNUA4TAJ2FSYO5S4NXECVMMMCXKFL5CAHPQBQJ6WREQ3YFC3Q, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.093051314 (58h26m8.964134359s) [noise=true , address=145.79.227.16:4933 , piece_id=LHF64L3SWXFVZ5X5D67I44WSJNKRFB5G4YP7DFHDWOAYE23II3OA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.098812273 (58h26m8.969895318s) [noise=true , address=145.79.227.18:4010 , piece_id=S2EOTEV4TBNNDOBN7TIX76AJE2WI5P5DM3JK2NJB25TISYLS7VSQ, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.102616718 (58h26m8.973699763s) [noise=true , address=145.79.227.25:4627 , piece_id=K7UYU3W2CACUAQUUL72GG5F7DP5CHAHZ4M47TZG6NBN5Z6ZCDRBA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.118902949 (58h26m8.989985994s) [noise=true , address=145.79.227.34:4928 , piece_id=DRKF2DRSCBAYSZO66A2ZON2JS6HIJIT2YUWFZRS3W2CKFTV3NLVA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.124783838 (58h26m8.995866883s) [noise=true , address=77.90.31.18:4825 , piece_id=EJ4LO4TCEFB4OQQLFOEUYGQKK462ZPXXEIIY5CHF2PERXWDGQNDA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.126718282 (58h26m8.997801327s) [noise=true , address=145.79.227.11:4740 , piece_id=KL54LL2UQRT6MH2K3NQQJB3BKPX4QRWQMXRK27MW2U7CHFDCCWFA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.145158485 (58h26m9.01624153s) [noise=true , address=77.90.31.10:4550 , piece_id=PDCZ7VXNHQJ6LYPG64GZRYP4MRBGVPXCWTXSRT3UV2N6HGMDE2KA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.156650047 (58h26m9.027733092s) [noise=true , address=145.79.227.15:4908 , piece_id=JIUZSMOAKBFCFAX6RQ52TCNZMYBREQCYOEM75U3H2QYR4WH3KWCA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.162020363 (58h26m9.033103398s) [noise=true , address=77.90.31.10:4539 , piece_id=27YFK55QLZDEULYABQ3OBDSPDDEIUJ2MJSD7X3A4K6HRFJMRQJUA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.198899907 (58h26m9.069982942s) [noise=true , address=77.90.31.12:4621 , piece_id=S4FVJHEPIO2MVO56ZRIJ27RMTV723MERPLJLVD2T2GAO3T7K24TA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.208468337 (58h26m9.079551382s) [noise=true , address=45.91.116.4:4945 , piece_id=6UNWQQAWWWWC2XVUCDRCTAYJDTHBXAZJMXA6KLCTOP5PYSAJCCGA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.212217697 (58h26m9.083300732s) [noise=true , address=45.91.116.12:4254 , piece_id=UCKU4TBXJC4PSYHRDPUXETXLAILDOWA5NN7RCVVNYLVUBY7FBIFA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.215832390 (58h26m9.086915435s) [noise=true , address=77.90.31.11:4579 , piece_id=JINRNEVUAP2VI5Y4CR7ALWDQIAIAFX3SP2VVZSXDDG6XPUKASE3Q, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.215862086 (58h26m9.086945161s) [seg_pos={0 0}, upload=51329]: Segment reached optimal threshold of 14 pieces. Dec 8 01:55:09.216078098 (58h26m9.087161133s) [noise=true , address=45.91.116.6:4003 , piece_id=CRONTV2B2AVZTLKU3YNIEGQZSON7XHHL7S3K6KNNJV4YR5FHQ3PA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1veqoBGw6a8AGMoigyuNJMjcEvCT93xRq2B1d2KNhs6aEmSqVF): protocol: expected piece hash; context canceled; EOF Dec 8 01:55:09.216085662 (58h26m9.087168707s) [noise=true , address=45.91.116.6:4003 , piece_id=CRONTV2B2AVZTLKU3YNIEGQZSON7XHHL7S3K6KNNJV4YR5FHQ3PA, seg_pos={0 0}, upload=51329]: Long tail cancellation due to optimal threshold Dec 8 01:55:09.216307705 (58h26m9.08739075s) [noise=true , address=145.79.227.30:4786 , piece_id=VM65RUN3GUZZFUOHFYE7ZLPWU6COCQUMY3IK7GCNWPIUZAMEBR7Q, seg_pos={0 0}, upload=51329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125TWU7R4pkmrUxsNLhgTD475aLbjUJ7psA6drJtE7oDqZXYJ4Q): protocol: expected piece hash; context canceled; EOF Dec 8 01:55:09.216313956 (58h26m9.087396991s) [noise=true , address=145.79.227.30:4786 , piece_id=VM65RUN3GUZZFUOHFYE7ZLPWU6COCQUMY3IK7GCNWPIUZAMEBR7Q, seg_pos={0 0}, upload=51329]: Long tail cancellation due to optimal threshold Dec 8 01:55:09.216414809 (58h26m9.087497844s) [noise=true , address=77.90.31.16:4760 , piece_id=Q67LYTDJCAR2TFVSAIIYCUYM7D44JFPBLCSKRNLXH26EAW5UQ3GA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1e6p1TD5CqUyWdZjkrxh2kRqMuyDiMUr9eimfeayDAKwhtBrZ8): protocol: expected piece hash; context canceled; EOF Dec 8 01:55:09.216420681 (58h26m9.087503726s) [noise=true , address=77.90.31.16:4760 , piece_id=Q67LYTDJCAR2TFVSAIIYCUYM7D44JFPBLCSKRNLXH26EAW5UQ3GA, seg_pos={0 0}, upload=51329]: Long tail cancellation due to optimal threshold Dec 8 01:55:09.216562861 (58h26m9.087645906s) [noise=true , address=45.91.116.7:4037 , piece_id=UBNFDSXQBISC5QQSNGAAZIR7LRE7FCYRVYN5MMOY3A2ALEOQBYQQ, seg_pos={0 0}, upload=51329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iNPwE2XSfXcaXAXcrH2jcyNGYeDiiWvzAcDQffrmjPwTVqVmK): protocol: expected piece hash; context canceled; EOF Dec 8 01:55:09.216570025 (58h26m9.08765307s) [noise=true , address=45.91.116.7:4037 , piece_id=UBNFDSXQBISC5QQSNGAAZIR7LRE7FCYRVYN5MMOY3A2ALEOQBYQQ, seg_pos={0 0}, upload=51329]: Long tail cancellation due to optimal threshold Dec 8 01:55:09.216585564 (58h26m9.087668609s) [seg_pos={0 0}, upload=51329]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 8 01:55:09.216654365 (58h26m9.0877374s) [upload=51329]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 01:55:09.216659064 (58h26m9.087742099s) [upload=51329]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 01:55:35.852213564 (58h26m35.723296629s) [upload=51329]: Done uploading segments... Dec 8 01:55:35.852273619 (58h26m35.723356664s) [upload=51329]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.307231002s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 5.746271228s trace: https://jaeger.dev.storj.io/trace/4030003299248e86 FAIL (2025-12-08 02:30:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.746291325s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 5.116741353s trace: https://jaeger.dev.storj.io/trace/7af096edea7ffcf5 FAIL (2025-12-08 02:30:45 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.116810455s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 5.225554763s trace: https://jaeger.dev.storj.io/trace/49d209695fd399e8 FAIL (2025-12-08 02:30:45 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 5.225596061s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........X.........X......................................] duration: 7.105206735s trace: https://jaeger.dev.storj.io/trace/e7164007ac0fd55 FAIL (2025-12-08 02:30:38 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 7.105233568s core uplink us1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 5.754377714s trace: https://jaeger.dev.storj.io/trace/49d209695fd399e8 FAIL (2025-12-08 02:30:30 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 5.754410766s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 6.252860652s trace: https://jaeger.dev.storj.io/trace/27d92c196f473eec FAIL (2025-12-08 02:30:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 4891s) [upload=51469]: Got next segment. Inline: false Dec 8 02:04:21.143134547 (58h35m21.014217592s) [upload=51469]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 02:04:21.143141259 (58h35m21.014224304s) [upload=51469]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 02:04:22.267502706 (58h35m22.138585751s) [seg_pos={0 0}, upload=51469]: Begin upload segment... Dec 8 02:04:22.267624127 (58h35m22.138707172s) [seg_pos={0 0}, upload=51469]: Done begin upload segment. Dec 8 02:04:22.267698027 (58h35m22.138781073s) [noise=true , address=77.90.31.11:4554 , piece_id=BQP7XTRA2SBVINT4J7JAOF3ORNENBBIQN4ABMCEMQQ6FQL6QUOKA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.267712997 (58h35m22.138796042s) [noise=true , address=145.79.227.41:4087 , piece_id=XO6K3CXK7KZ5VEVXULHKQXLSSLIIWIX3VKQOSD7HY25WMNMCDLEQ, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.268434680 (58h35m22.139517715s) [noise=true , address=145.79.227.16:4910 , piece_id=ETE7ZBHT5IJUOBWCA5BGLKC2ANDEUB2M3DOMXWWS2RBWSJQBARRQ, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.268550331 (58h35m22.139633366s) [noise=true , address=145.79.227.26:4653 , piece_id=DKW72PEWPIOHPYHLI54PBU7NR3HGUBLFM3MN4XQQFWIIZ4FHIUNA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.268913111 (58h35m22.139996146s) [noise=true , address=145.79.227.23:4545 , piece_id=NBYCSU2KSHTAL223AHKHREQWKXUEQKJU46TY3SPYTKT5N5T2OUSA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.268924032 (58h35m22.140007067s) [noise=true , address=145.79.227.40:4051 , piece_id=7CQTGHFZ542WUHYX2CTOPN4PF2KDOOLDGV64THDBZCYOCXA3JWDA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.269398886 (58h35m22.140481921s) [noise=true , address=77.90.31.10:4518 , piece_id=6AR3NAFODP54SKG3SJSJSIQQFOYXO2C65WVNN4Z4XG426ZEOEDQA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.269414686 (58h35m22.140497721s) [noise=true , address=145.79.227.25:4619 , piece_id=XBNGVRIQLQUWKKBF4ULL7ZJDDT75QLY7ES2J7WHYWRD2ZLHOGU2Q, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.269753261 (58h35m22.140836296s) [noise=true , address=145.79.227.23:4547 , piece_id=5Y6V5QWLCWNOLT2GDNUEIRSG75KGNGIFLBCBGSIT7XH46RZYVDGQ, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.270016322 (58h35m22.141099367s) [noise=true , address=77.90.31.13:4628 , piece_id=LWTOCQRRA7HLI5ZPPRQ25IM22LMQNRR5H3TTOTSTGCFOYFZMABAA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.270259175 (58h35m22.14134222s) [noise=true , address=145.79.227.25:4617 , piece_id=S4ETCJ6WMCPXXQCYPPBBMHLMK2SVJV4S5VOKHGTDAGSGG6WRYH3Q, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.270655339 (58h35m22.141738374s) [noise=true , address=145.79.227.38:4977 , piece_id=73WXA33TWM4MFVTHDHW3PXJVABMCVMQYI3P42AHT3OYPKMXIVTUQ, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.271143158 (58h35m22.142246412s) [noise=true , address=145.79.227.28:4727 , piece_id=UYRK457JBF2AHCIXZ4XOIL7YHOVYZ7IPDUTB6WOZZUDEW5Q7NNCQ, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.271230565 (58h35m22.14231361s) [noise=true , address=145.79.227.39:4013 , piece_id=UI22RDXJ2DY3RFNSJVNT43XQJ6B2S3EO6SFLYHXW4VPVQJSYYF6Q, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.271662086 (58h35m22.142745121s) [noise=true , address=145.79.227.22:4511 , piece_id=GGA463DAOSZCW7SQGZW2LGFRQVNUOUJ2L2GXRXLSDPC65KVPY3TA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.272592658 (58h35m22.143675703s) [noise=true , address=145.79.227.24:4583 , piece_id=WYTXGXG77ZW4RGWX7YWDU2NH6R5CCPVBPQH3ISHUAJVKZJRONS5Q, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:24.009199123 (58h35m23.880282158s) [upload=51469]: Next returned nil segment Dec 8 02:04:24.009221235 (58h35m23.88030428s) [upload=51469]: Waiting for error group managing segments... Dec 8 02:04:24.293906896 (58h35m24.164989931s) [noise=true , address=145.79.227.24:4583 , piece_id=WYTXGXG77ZW4RGWX7YWDU2NH6R5CCPVBPQH3ISHUAJVKZJRONS5Q, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.298576738 (58h35m24.169659783s) [noise=true , address=145.79.227.28:4727 , piece_id=UYRK457JBF2AHCIXZ4XOIL7YHOVYZ7IPDUTB6WOZZUDEW5Q7NNCQ, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.301356571 (58h35m24.172439616s) [noise=true , address=145.79.227.25:4619 , piece_id=XBNGVRIQLQUWKKBF4ULL7ZJDDT75QLY7ES2J7WHYWRD2ZLHOGU2Q, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.326466765 (58h35m24.19754981s) [noise=true , address=145.79.227.16:4910 , piece_id=ETE7ZBHT5IJUOBWCA5BGLKC2ANDEUB2M3DOMXWWS2RBWSJQBARRQ, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.328633530 (58h35m24.199716586s) [noise=true , address=145.79.227.26:4653 , piece_id=DKW72PEWPIOHPYHLI54PBU7NR3HGUBLFM3MN4XQQFWIIZ4FHIUNA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.360508106 (58h35m24.231591151s) [noise=true , address=145.79.227.40:4051 , piece_id=7CQTGHFZ542WUHYX2CTOPN4PF2KDOOLDGV64THDBZCYOCXA3JWDA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.361116263 (58h35m24.232199308s) [noise=true , address=145.79.227.25:4617 , piece_id=S4ETCJ6WMCPXXQCYPPBBMHLMK2SVJV4S5VOKHGTDAGSGG6WRYH3Q, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.361695086 (58h35m24.232778131s) [noise=true , address=145.79.227.41:4087 , piece_id=XO6K3CXK7KZ5VEVXULHKQXLSSLIIWIX3VKQOSD7HY25WMNMCDLEQ, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.371520525 (58h35m24.24260356s) [noise=true , address=145.79.227.38:4977 , piece_id=73WXA33TWM4MFVTHDHW3PXJVABMCVMQYI3P42AHT3OYPKMXIVTUQ, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.381271893 (58h35m24.252354928s) [noise=true , address=145.79.227.23:4545 , piece_id=NBYCSU2KSHTAL223AHKHREQWKXUEQKJU46TY3SPYTKT5N5T2OUSA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.386403053 (58h35m24.257486098s) [noise=true , address=77.90.31.11:4554 , piece_id=BQP7XTRA2SBVINT4J7JAOF3ORNENBBIQN4ABMCEMQQ6FQL6QUOKA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.386427209 (58h35m24.257510254s) [noise=true , address=77.90.31.10:4518 , piece_id=6AR3NAFODP54SKG3SJSJSIQQFOYXO2C65WVNN4Z4XG426ZEOEDQA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.400489386 (58h35m24.271572431s) [noise=true , address=77.90.31.13:4628 , piece_id=LWTOCQRRA7HLI5ZPPRQ25IM22LMQNRR5H3TTOTSTGCFOYFZMABAA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.400513903 (58h35m24.271596938s) [seg_pos={0 0}, upload=51469]: Segment reached optimal threshold of 13 pieces. Dec 8 02:04:24.400813233 (58h35m24.271896288s) [noise=true , address=145.79.227.39:4013 , piece_id=UI22RDXJ2DY3RFNSJVNT43XQJ6B2S3EO6SFLYHXW4VPVQJSYYF6Q, seg_pos={0 0}, upload=51469]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b): protocol: write tcp 188.166.225.170:39434->145.79.227.39:4013: use of closed network connection; write tcp 188.166.225.170:39434->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39434->145.79.227.39:4013: use of closed network connection Dec 8 02:04:24.400825677 (58h35m24.271908722s) [noise=true , address=145.79.227.39:4013 , piece_id=UI22RDXJ2DY3RFNSJVNT43XQJ6B2S3EO6SFLYHXW4VPVQJSYYF6Q, seg_pos={0 0}, upload=51469]: Long tail cancellation due to optimal threshold Dec 8 02:04:24.400873027 (58h35m24.271956082s) [noise=true , address=145.79.227.22:4511 , piece_id=GGA463DAOSZCW7SQGZW2LGFRQVNUOUJ2L2GXRXLSDPC65KVPY3TA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG): protocol: write tcp 188.166.225.170:47926->145.79.227.22:4511: use of closed network connection; write tcp 188.166.225.170:47926->145.79.227.22:4511: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47926->145.79.227.22:4511: use of closed network connection Dec 8 02:04:24.400894248 (58h35m24.271977293s) [noise=true , address=145.79.227.22:4511 , piece_id=GGA463DAOSZCW7SQGZW2LGFRQVNUOUJ2L2GXRXLSDPC65KVPY3TA, seg_pos={0 0}, upload=51469]: Long tail cancellation due to optimal threshold Dec 8 02:04:24.401247129 (58h35m24.272330184s) [noise=true , address=145.79.227.23:4547 , piece_id=5Y6V5QWLCWNOLT2GDNUEIRSG75KGNGIFLBCBGSIT7XH46RZYVDGQ, seg_pos={0 0}, upload=51469]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW): protocol: read tcp 188.166.225.170:43344->145.79.227.23:4547: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 02:04:24.401260003 (58h35m24.272343059s) [noise=true , address=145.79.227.23:4547 , piece_id=5Y6V5QWLCWNOLT2GDNUEIRSG75KGNGIFLBCBGSIT7XH46RZYVDGQ, seg_pos={0 0}, upload=51469]: Long tail cancellation due to optimal threshold Dec 8 02:04:24.401285452 (58h35m24.272368497s) [seg_pos={0 0}, upload=51469]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 02:04:24.401382247 (58h35m24.272465292s) [upload=51469]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 02:04:24.401391394 (58h35m24.272474429s) [upload=51469]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 02:05:39.049922629 (58h36m38.921005704s) [upload=51469]: Done uploading segments... Dec 8 02:05:39.050003342 (58h36m38.921086377s) [upload=51469]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.252890388s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 6.424191576s trace: https://jaeger.dev.storj.io/trace/57258282a8e974fd FAIL (2025-12-08 02:30:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== UA4TAJ2FSYO5S4NXECVMMMCXKFL5CAHPQBQJ6WREQ3YFC3Q, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.202906305 (58h26m7.07398934s) [noise=true , address=45.91.116.4:4945 , piece_id=6UNWQQAWWWWC2XVUCDRCTAYJDTHBXAZJMXA6KLCTOP5PYSAJCCGA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.203371201 (58h26m7.074454246s) [noise=true , address=145.79.227.11:4740 , piece_id=KL54LL2UQRT6MH2K3NQQJB3BKPX4QRWQMXRK27MW2U7CHFDCCWFA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.203861354 (58h26m7.07494439s) [noise=true , address=77.90.31.12:4621 , piece_id=S4FVJHEPIO2MVO56ZRIJ27RMTV723MERPLJLVD2T2GAO3T7K24TA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.204290401 (58h26m7.075373446s) [noise=true , address=145.79.227.34:4928 , piece_id=DRKF2DRSCBAYSZO66A2ZON2JS6HIJIT2YUWFZRS3W2CKFTV3NLVA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.204711363 (58h26m7.075794398s) [noise=true , address=145.79.227.15:4908 , piece_id=JIUZSMOAKBFCFAX6RQ52TCNZMYBREQCYOEM75U3H2QYR4WH3KWCA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.205118208 (58h26m7.076201253s) [noise=true , address=77.90.31.18:4825 , piece_id=EJ4LO4TCEFB4OQQLFOEUYGQKK462ZPXXEIIY5CHF2PERXWDGQNDA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.205554277 (58h26m7.076637323s) [noise=true , address=145.79.227.30:4786 , piece_id=VM65RUN3GUZZFUOHFYE7ZLPWU6COCQUMY3IK7GCNWPIUZAMEBR7Q, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.206116137 (58h26m7.077199182s) [noise=true , address=145.79.227.25:4627 , piece_id=K7UYU3W2CACUAQUUL72GG5F7DP5CHAHZ4M47TZG6NBN5Z6ZCDRBA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.206664562 (58h26m7.077747607s) [noise=true , address=77.90.31.16:4760 , piece_id=Q67LYTDJCAR2TFVSAIIYCUYM7D44JFPBLCSKRNLXH26EAW5UQ3GA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.207102556 (58h26m7.078185601s) [noise=true , address=45.91.116.7:4037 , piece_id=UBNFDSXQBISC5QQSNGAAZIR7LRE7FCYRVYN5MMOY3A2ALEOQBYQQ, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.207519911 (58h26m7.078602956s) [noise=true , address=77.90.31.11:4579 , piece_id=JINRNEVUAP2VI5Y4CR7ALWDQIAIAFX3SP2VVZSXDDG6XPUKASE3Q, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.207967393 (58h26m7.079050438s) [noise=true , address=77.90.31.10:4550 , piece_id=PDCZ7VXNHQJ6LYPG64GZRYP4MRBGVPXCWTXSRT3UV2N6HGMDE2KA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.208416427 (58h26m7.079499472s) [noise=true , address=45.91.116.6:4003 , piece_id=CRONTV2B2AVZTLKU3YNIEGQZSON7XHHL7S3K6KNNJV4YR5FHQ3PA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.208830746 (58h26m7.079913791s) [noise=true , address=45.91.116.12:4254 , piece_id=UCKU4TBXJC4PSYHRDPUXETXLAILDOWA5NN7RCVVNYLVUBY7FBIFA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.209255245 (58h26m7.080338291s) [noise=true , address=145.79.227.16:4933 , piece_id=LHF64L3SWXFVZ5X5D67I44WSJNKRFB5G4YP7DFHDWOAYE23II3OA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.209662921 (58h26m7.080745966s) [noise=true , address=145.79.227.18:4010 , piece_id=S2EOTEV4TBNNDOBN7TIX76AJE2WI5P5DM3JK2NJB25TISYLS7VSQ, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.214493680 (58h26m7.085576725s) [noise=true , address=77.90.31.10:4539 , piece_id=27YFK55QLZDEULYABQ3OBDSPDDEIUJ2MJSD7X3A4K6HRFJMRQJUA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:08.555121654 (58h26m8.426204699s) [upload=51329]: Next returned nil segment Dec 8 01:55:08.555145519 (58h26m8.426228564s) [upload=51329]: Waiting for error group managing segments... Dec 8 01:55:08.935193756 (58h26m8.806276791s) [noise=true , address=64.49.13.27:4957 , piece_id=7METNUA4TAJ2FSYO5S4NXECVMMMCXKFL5CAHPQBQJ6WREQ3YFC3Q, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.093051314 (58h26m8.964134359s) [noise=true , address=145.79.227.16:4933 , piece_id=LHF64L3SWXFVZ5X5D67I44WSJNKRFB5G4YP7DFHDWOAYE23II3OA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.098812273 (58h26m8.969895318s) [noise=true , address=145.79.227.18:4010 , piece_id=S2EOTEV4TBNNDOBN7TIX76AJE2WI5P5DM3JK2NJB25TISYLS7VSQ, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.102616718 (58h26m8.973699763s) [noise=true , address=145.79.227.25:4627 , piece_id=K7UYU3W2CACUAQUUL72GG5F7DP5CHAHZ4M47TZG6NBN5Z6ZCDRBA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.118902949 (58h26m8.989985994s) [noise=true , address=145.79.227.34:4928 , piece_id=DRKF2DRSCBAYSZO66A2ZON2JS6HIJIT2YUWFZRS3W2CKFTV3NLVA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.124783838 (58h26m8.995866883s) [noise=true , address=77.90.31.18:4825 , piece_id=EJ4LO4TCEFB4OQQLFOEUYGQKK462ZPXXEIIY5CHF2PERXWDGQNDA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.126718282 (58h26m8.997801327s) [noise=true , address=145.79.227.11:4740 , piece_id=KL54LL2UQRT6MH2K3NQQJB3BKPX4QRWQMXRK27MW2U7CHFDCCWFA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.145158485 (58h26m9.01624153s) [noise=true , address=77.90.31.10:4550 , piece_id=PDCZ7VXNHQJ6LYPG64GZRYP4MRBGVPXCWTXSRT3UV2N6HGMDE2KA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.156650047 (58h26m9.027733092s) [noise=true , address=145.79.227.15:4908 , piece_id=JIUZSMOAKBFCFAX6RQ52TCNZMYBREQCYOEM75U3H2QYR4WH3KWCA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.162020363 (58h26m9.033103398s) [noise=true , address=77.90.31.10:4539 , piece_id=27YFK55QLZDEULYABQ3OBDSPDDEIUJ2MJSD7X3A4K6HRFJMRQJUA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.198899907 (58h26m9.069982942s) [noise=true , address=77.90.31.12:4621 , piece_id=S4FVJHEPIO2MVO56ZRIJ27RMTV723MERPLJLVD2T2GAO3T7K24TA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.208468337 (58h26m9.079551382s) [noise=true , address=45.91.116.4:4945 , piece_id=6UNWQQAWWWWC2XVUCDRCTAYJDTHBXAZJMXA6KLCTOP5PYSAJCCGA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.212217697 (58h26m9.083300732s) [noise=true , address=45.91.116.12:4254 , piece_id=UCKU4TBXJC4PSYHRDPUXETXLAILDOWA5NN7RCVVNYLVUBY7FBIFA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.215832390 (58h26m9.086915435s) [noise=true , address=77.90.31.11:4579 , piece_id=JINRNEVUAP2VI5Y4CR7ALWDQIAIAFX3SP2VVZSXDDG6XPUKASE3Q, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.215862086 (58h26m9.086945161s) [seg_pos={0 0}, upload=51329]: Segment reached optimal threshold of 14 pieces. Dec 8 01:55:09.216078098 (58h26m9.087161133s) [noise=true , address=45.91.116.6:4003 , piece_id=CRONTV2B2AVZTLKU3YNIEGQZSON7XHHL7S3K6KNNJV4YR5FHQ3PA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1veqoBGw6a8AGMoigyuNJMjcEvCT93xRq2B1d2KNhs6aEmSqVF): protocol: expected piece hash; context canceled; EOF Dec 8 01:55:09.216085662 (58h26m9.087168707s) [noise=true , address=45.91.116.6:4003 , piece_id=CRONTV2B2AVZTLKU3YNIEGQZSON7XHHL7S3K6KNNJV4YR5FHQ3PA, seg_pos={0 0}, upload=51329]: Long tail cancellation due to optimal threshold Dec 8 01:55:09.216307705 (58h26m9.08739075s) [noise=true , address=145.79.227.30:4786 , piece_id=VM65RUN3GUZZFUOHFYE7ZLPWU6COCQUMY3IK7GCNWPIUZAMEBR7Q, seg_pos={0 0}, upload=51329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125TWU7R4pkmrUxsNLhgTD475aLbjUJ7psA6drJtE7oDqZXYJ4Q): protocol: expected piece hash; context canceled; EOF Dec 8 01:55:09.216313956 (58h26m9.087396991s) [noise=true , address=145.79.227.30:4786 , piece_id=VM65RUN3GUZZFUOHFYE7ZLPWU6COCQUMY3IK7GCNWPIUZAMEBR7Q, seg_pos={0 0}, upload=51329]: Long tail cancellation due to optimal threshold Dec 8 01:55:09.216414809 (58h26m9.087497844s) [noise=true , address=77.90.31.16:4760 , piece_id=Q67LYTDJCAR2TFVSAIIYCUYM7D44JFPBLCSKRNLXH26EAW5UQ3GA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1e6p1TD5CqUyWdZjkrxh2kRqMuyDiMUr9eimfeayDAKwhtBrZ8): protocol: expected piece hash; context canceled; EOF Dec 8 01:55:09.216420681 (58h26m9.087503726s) [noise=true , address=77.90.31.16:4760 , piece_id=Q67LYTDJCAR2TFVSAIIYCUYM7D44JFPBLCSKRNLXH26EAW5UQ3GA, seg_pos={0 0}, upload=51329]: Long tail cancellation due to optimal threshold Dec 8 01:55:09.216562861 (58h26m9.087645906s) [noise=true , address=45.91.116.7:4037 , piece_id=UBNFDSXQBISC5QQSNGAAZIR7LRE7FCYRVYN5MMOY3A2ALEOQBYQQ, seg_pos={0 0}, upload=51329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iNPwE2XSfXcaXAXcrH2jcyNGYeDiiWvzAcDQffrmjPwTVqVmK): protocol: expected piece hash; context canceled; EOF Dec 8 01:55:09.216570025 (58h26m9.08765307s) [noise=true , address=45.91.116.7:4037 , piece_id=UBNFDSXQBISC5QQSNGAAZIR7LRE7FCYRVYN5MMOY3A2ALEOQBYQQ, seg_pos={0 0}, upload=51329]: Long tail cancellation due to optimal threshold Dec 8 01:55:09.216585564 (58h26m9.087668609s) [seg_pos={0 0}, upload=51329]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 8 01:55:09.216654365 (58h26m9.0877374s) [upload=51329]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 01:55:09.216659064 (58h26m9.087742099s) [upload=51329]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 01:55:35.852213564 (58h26m35.723296629s) [upload=51329]: Done uploading segments... Dec 8 01:55:35.852273619 (58h26m35.723356664s) [upload=51329]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.424222645s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)