Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 3 out of 60 is failed (threshold=3) [X................X.........X................................] duration: 6.20846204s trace: https://jaeger.dev.storj.io/trace/e7164007ac0fd55 FAIL (2025-12-08 02:26:03 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 6.208494361s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 5.81532613s trace: https://jaeger.dev.storj.io/trace/4030003299248e86 FAIL (2025-12-08 02:25:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.815346038s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 6.094314012s trace: https://jaeger.dev.storj.io/trace/27d92c196f473eec FAIL (2025-12-08 02:25:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 4891s) [upload=51469]: Got next segment. Inline: false Dec 8 02:04:21.143134547 (58h35m21.014217592s) [upload=51469]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 02:04:21.143141259 (58h35m21.014224304s) [upload=51469]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 02:04:22.267502706 (58h35m22.138585751s) [seg_pos={0 0}, upload=51469]: Begin upload segment... Dec 8 02:04:22.267624127 (58h35m22.138707172s) [seg_pos={0 0}, upload=51469]: Done begin upload segment. Dec 8 02:04:22.267698027 (58h35m22.138781073s) [noise=true , address=77.90.31.11:4554 , piece_id=BQP7XTRA2SBVINT4J7JAOF3ORNENBBIQN4ABMCEMQQ6FQL6QUOKA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.267712997 (58h35m22.138796042s) [noise=true , address=145.79.227.41:4087 , piece_id=XO6K3CXK7KZ5VEVXULHKQXLSSLIIWIX3VKQOSD7HY25WMNMCDLEQ, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.268434680 (58h35m22.139517715s) [noise=true , address=145.79.227.16:4910 , piece_id=ETE7ZBHT5IJUOBWCA5BGLKC2ANDEUB2M3DOMXWWS2RBWSJQBARRQ, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.268550331 (58h35m22.139633366s) [noise=true , address=145.79.227.26:4653 , piece_id=DKW72PEWPIOHPYHLI54PBU7NR3HGUBLFM3MN4XQQFWIIZ4FHIUNA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.268913111 (58h35m22.139996146s) [noise=true , address=145.79.227.23:4545 , piece_id=NBYCSU2KSHTAL223AHKHREQWKXUEQKJU46TY3SPYTKT5N5T2OUSA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.268924032 (58h35m22.140007067s) [noise=true , address=145.79.227.40:4051 , piece_id=7CQTGHFZ542WUHYX2CTOPN4PF2KDOOLDGV64THDBZCYOCXA3JWDA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.269398886 (58h35m22.140481921s) [noise=true , address=77.90.31.10:4518 , piece_id=6AR3NAFODP54SKG3SJSJSIQQFOYXO2C65WVNN4Z4XG426ZEOEDQA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.269414686 (58h35m22.140497721s) [noise=true , address=145.79.227.25:4619 , piece_id=XBNGVRIQLQUWKKBF4ULL7ZJDDT75QLY7ES2J7WHYWRD2ZLHOGU2Q, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.269753261 (58h35m22.140836296s) [noise=true , address=145.79.227.23:4547 , piece_id=5Y6V5QWLCWNOLT2GDNUEIRSG75KGNGIFLBCBGSIT7XH46RZYVDGQ, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.270016322 (58h35m22.141099367s) [noise=true , address=77.90.31.13:4628 , piece_id=LWTOCQRRA7HLI5ZPPRQ25IM22LMQNRR5H3TTOTSTGCFOYFZMABAA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.270259175 (58h35m22.14134222s) [noise=true , address=145.79.227.25:4617 , piece_id=S4ETCJ6WMCPXXQCYPPBBMHLMK2SVJV4S5VOKHGTDAGSGG6WRYH3Q, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.270655339 (58h35m22.141738374s) [noise=true , address=145.79.227.38:4977 , piece_id=73WXA33TWM4MFVTHDHW3PXJVABMCVMQYI3P42AHT3OYPKMXIVTUQ, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.271143158 (58h35m22.142246412s) [noise=true , address=145.79.227.28:4727 , piece_id=UYRK457JBF2AHCIXZ4XOIL7YHOVYZ7IPDUTB6WOZZUDEW5Q7NNCQ, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.271230565 (58h35m22.14231361s) [noise=true , address=145.79.227.39:4013 , piece_id=UI22RDXJ2DY3RFNSJVNT43XQJ6B2S3EO6SFLYHXW4VPVQJSYYF6Q, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.271662086 (58h35m22.142745121s) [noise=true , address=145.79.227.22:4511 , piece_id=GGA463DAOSZCW7SQGZW2LGFRQVNUOUJ2L2GXRXLSDPC65KVPY3TA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.272592658 (58h35m22.143675703s) [noise=true , address=145.79.227.24:4583 , piece_id=WYTXGXG77ZW4RGWX7YWDU2NH6R5CCPVBPQH3ISHUAJVKZJRONS5Q, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:24.009199123 (58h35m23.880282158s) [upload=51469]: Next returned nil segment Dec 8 02:04:24.009221235 (58h35m23.88030428s) [upload=51469]: Waiting for error group managing segments... Dec 8 02:04:24.293906896 (58h35m24.164989931s) [noise=true , address=145.79.227.24:4583 , piece_id=WYTXGXG77ZW4RGWX7YWDU2NH6R5CCPVBPQH3ISHUAJVKZJRONS5Q, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.298576738 (58h35m24.169659783s) [noise=true , address=145.79.227.28:4727 , piece_id=UYRK457JBF2AHCIXZ4XOIL7YHOVYZ7IPDUTB6WOZZUDEW5Q7NNCQ, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.301356571 (58h35m24.172439616s) [noise=true , address=145.79.227.25:4619 , piece_id=XBNGVRIQLQUWKKBF4ULL7ZJDDT75QLY7ES2J7WHYWRD2ZLHOGU2Q, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.326466765 (58h35m24.19754981s) [noise=true , address=145.79.227.16:4910 , piece_id=ETE7ZBHT5IJUOBWCA5BGLKC2ANDEUB2M3DOMXWWS2RBWSJQBARRQ, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.328633530 (58h35m24.199716586s) [noise=true , address=145.79.227.26:4653 , piece_id=DKW72PEWPIOHPYHLI54PBU7NR3HGUBLFM3MN4XQQFWIIZ4FHIUNA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.360508106 (58h35m24.231591151s) [noise=true , address=145.79.227.40:4051 , piece_id=7CQTGHFZ542WUHYX2CTOPN4PF2KDOOLDGV64THDBZCYOCXA3JWDA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.361116263 (58h35m24.232199308s) [noise=true , address=145.79.227.25:4617 , piece_id=S4ETCJ6WMCPXXQCYPPBBMHLMK2SVJV4S5VOKHGTDAGSGG6WRYH3Q, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.361695086 (58h35m24.232778131s) [noise=true , address=145.79.227.41:4087 , piece_id=XO6K3CXK7KZ5VEVXULHKQXLSSLIIWIX3VKQOSD7HY25WMNMCDLEQ, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.371520525 (58h35m24.24260356s) [noise=true , address=145.79.227.38:4977 , piece_id=73WXA33TWM4MFVTHDHW3PXJVABMCVMQYI3P42AHT3OYPKMXIVTUQ, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.381271893 (58h35m24.252354928s) [noise=true , address=145.79.227.23:4545 , piece_id=NBYCSU2KSHTAL223AHKHREQWKXUEQKJU46TY3SPYTKT5N5T2OUSA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.386403053 (58h35m24.257486098s) [noise=true , address=77.90.31.11:4554 , piece_id=BQP7XTRA2SBVINT4J7JAOF3ORNENBBIQN4ABMCEMQQ6FQL6QUOKA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.386427209 (58h35m24.257510254s) [noise=true , address=77.90.31.10:4518 , piece_id=6AR3NAFODP54SKG3SJSJSIQQFOYXO2C65WVNN4Z4XG426ZEOEDQA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.400489386 (58h35m24.271572431s) [noise=true , address=77.90.31.13:4628 , piece_id=LWTOCQRRA7HLI5ZPPRQ25IM22LMQNRR5H3TTOTSTGCFOYFZMABAA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.400513903 (58h35m24.271596938s) [seg_pos={0 0}, upload=51469]: Segment reached optimal threshold of 13 pieces. Dec 8 02:04:24.400813233 (58h35m24.271896288s) [noise=true , address=145.79.227.39:4013 , piece_id=UI22RDXJ2DY3RFNSJVNT43XQJ6B2S3EO6SFLYHXW4VPVQJSYYF6Q, seg_pos={0 0}, upload=51469]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b): protocol: write tcp 188.166.225.170:39434->145.79.227.39:4013: use of closed network connection; write tcp 188.166.225.170:39434->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39434->145.79.227.39:4013: use of closed network connection Dec 8 02:04:24.400825677 (58h35m24.271908722s) [noise=true , address=145.79.227.39:4013 , piece_id=UI22RDXJ2DY3RFNSJVNT43XQJ6B2S3EO6SFLYHXW4VPVQJSYYF6Q, seg_pos={0 0}, upload=51469]: Long tail cancellation due to optimal threshold Dec 8 02:04:24.400873027 (58h35m24.271956082s) [noise=true , address=145.79.227.22:4511 , piece_id=GGA463DAOSZCW7SQGZW2LGFRQVNUOUJ2L2GXRXLSDPC65KVPY3TA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG): protocol: write tcp 188.166.225.170:47926->145.79.227.22:4511: use of closed network connection; write tcp 188.166.225.170:47926->145.79.227.22:4511: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47926->145.79.227.22:4511: use of closed network connection Dec 8 02:04:24.400894248 (58h35m24.271977293s) [noise=true , address=145.79.227.22:4511 , piece_id=GGA463DAOSZCW7SQGZW2LGFRQVNUOUJ2L2GXRXLSDPC65KVPY3TA, seg_pos={0 0}, upload=51469]: Long tail cancellation due to optimal threshold Dec 8 02:04:24.401247129 (58h35m24.272330184s) [noise=true , address=145.79.227.23:4547 , piece_id=5Y6V5QWLCWNOLT2GDNUEIRSG75KGNGIFLBCBGSIT7XH46RZYVDGQ, seg_pos={0 0}, upload=51469]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW): protocol: read tcp 188.166.225.170:43344->145.79.227.23:4547: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 02:04:24.401260003 (58h35m24.272343059s) [noise=true , address=145.79.227.23:4547 , piece_id=5Y6V5QWLCWNOLT2GDNUEIRSG75KGNGIFLBCBGSIT7XH46RZYVDGQ, seg_pos={0 0}, upload=51469]: Long tail cancellation due to optimal threshold Dec 8 02:04:24.401285452 (58h35m24.272368497s) [seg_pos={0 0}, upload=51469]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 02:04:24.401382247 (58h35m24.272465292s) [upload=51469]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 02:04:24.401391394 (58h35m24.272474429s) [upload=51469]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 02:05:39.049922629 (58h36m38.921005704s) [upload=51469]: Done uploading segments... Dec 8 02:05:39.050003342 (58h36m38.921086377s) [upload=51469]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.094381429s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 15.795449287s trace: https://jaeger.dev.storj.io/trace/57258282a8e974fd FAIL (2025-12-08 02:25:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== UA4TAJ2FSYO5S4NXECVMMMCXKFL5CAHPQBQJ6WREQ3YFC3Q, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.202906305 (58h26m7.07398934s) [noise=true , address=45.91.116.4:4945 , piece_id=6UNWQQAWWWWC2XVUCDRCTAYJDTHBXAZJMXA6KLCTOP5PYSAJCCGA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.203371201 (58h26m7.074454246s) [noise=true , address=145.79.227.11:4740 , piece_id=KL54LL2UQRT6MH2K3NQQJB3BKPX4QRWQMXRK27MW2U7CHFDCCWFA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.203861354 (58h26m7.07494439s) [noise=true , address=77.90.31.12:4621 , piece_id=S4FVJHEPIO2MVO56ZRIJ27RMTV723MERPLJLVD2T2GAO3T7K24TA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.204290401 (58h26m7.075373446s) [noise=true , address=145.79.227.34:4928 , piece_id=DRKF2DRSCBAYSZO66A2ZON2JS6HIJIT2YUWFZRS3W2CKFTV3NLVA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.204711363 (58h26m7.075794398s) [noise=true , address=145.79.227.15:4908 , piece_id=JIUZSMOAKBFCFAX6RQ52TCNZMYBREQCYOEM75U3H2QYR4WH3KWCA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.205118208 (58h26m7.076201253s) [noise=true , address=77.90.31.18:4825 , piece_id=EJ4LO4TCEFB4OQQLFOEUYGQKK462ZPXXEIIY5CHF2PERXWDGQNDA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.205554277 (58h26m7.076637323s) [noise=true , address=145.79.227.30:4786 , piece_id=VM65RUN3GUZZFUOHFYE7ZLPWU6COCQUMY3IK7GCNWPIUZAMEBR7Q, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.206116137 (58h26m7.077199182s) [noise=true , address=145.79.227.25:4627 , piece_id=K7UYU3W2CACUAQUUL72GG5F7DP5CHAHZ4M47TZG6NBN5Z6ZCDRBA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.206664562 (58h26m7.077747607s) [noise=true , address=77.90.31.16:4760 , piece_id=Q67LYTDJCAR2TFVSAIIYCUYM7D44JFPBLCSKRNLXH26EAW5UQ3GA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.207102556 (58h26m7.078185601s) [noise=true , address=45.91.116.7:4037 , piece_id=UBNFDSXQBISC5QQSNGAAZIR7LRE7FCYRVYN5MMOY3A2ALEOQBYQQ, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.207519911 (58h26m7.078602956s) [noise=true , address=77.90.31.11:4579 , piece_id=JINRNEVUAP2VI5Y4CR7ALWDQIAIAFX3SP2VVZSXDDG6XPUKASE3Q, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.207967393 (58h26m7.079050438s) [noise=true , address=77.90.31.10:4550 , piece_id=PDCZ7VXNHQJ6LYPG64GZRYP4MRBGVPXCWTXSRT3UV2N6HGMDE2KA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.208416427 (58h26m7.079499472s) [noise=true , address=45.91.116.6:4003 , piece_id=CRONTV2B2AVZTLKU3YNIEGQZSON7XHHL7S3K6KNNJV4YR5FHQ3PA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.208830746 (58h26m7.079913791s) [noise=true , address=45.91.116.12:4254 , piece_id=UCKU4TBXJC4PSYHRDPUXETXLAILDOWA5NN7RCVVNYLVUBY7FBIFA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.209255245 (58h26m7.080338291s) [noise=true , address=145.79.227.16:4933 , piece_id=LHF64L3SWXFVZ5X5D67I44WSJNKRFB5G4YP7DFHDWOAYE23II3OA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.209662921 (58h26m7.080745966s) [noise=true , address=145.79.227.18:4010 , piece_id=S2EOTEV4TBNNDOBN7TIX76AJE2WI5P5DM3JK2NJB25TISYLS7VSQ, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.214493680 (58h26m7.085576725s) [noise=true , address=77.90.31.10:4539 , piece_id=27YFK55QLZDEULYABQ3OBDSPDDEIUJ2MJSD7X3A4K6HRFJMRQJUA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:08.555121654 (58h26m8.426204699s) [upload=51329]: Next returned nil segment Dec 8 01:55:08.555145519 (58h26m8.426228564s) [upload=51329]: Waiting for error group managing segments... Dec 8 01:55:08.935193756 (58h26m8.806276791s) [noise=true , address=64.49.13.27:4957 , piece_id=7METNUA4TAJ2FSYO5S4NXECVMMMCXKFL5CAHPQBQJ6WREQ3YFC3Q, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.093051314 (58h26m8.964134359s) [noise=true , address=145.79.227.16:4933 , piece_id=LHF64L3SWXFVZ5X5D67I44WSJNKRFB5G4YP7DFHDWOAYE23II3OA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.098812273 (58h26m8.969895318s) [noise=true , address=145.79.227.18:4010 , piece_id=S2EOTEV4TBNNDOBN7TIX76AJE2WI5P5DM3JK2NJB25TISYLS7VSQ, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.102616718 (58h26m8.973699763s) [noise=true , address=145.79.227.25:4627 , piece_id=K7UYU3W2CACUAQUUL72GG5F7DP5CHAHZ4M47TZG6NBN5Z6ZCDRBA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.118902949 (58h26m8.989985994s) [noise=true , address=145.79.227.34:4928 , piece_id=DRKF2DRSCBAYSZO66A2ZON2JS6HIJIT2YUWFZRS3W2CKFTV3NLVA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.124783838 (58h26m8.995866883s) [noise=true , address=77.90.31.18:4825 , piece_id=EJ4LO4TCEFB4OQQLFOEUYGQKK462ZPXXEIIY5CHF2PERXWDGQNDA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.126718282 (58h26m8.997801327s) [noise=true , address=145.79.227.11:4740 , piece_id=KL54LL2UQRT6MH2K3NQQJB3BKPX4QRWQMXRK27MW2U7CHFDCCWFA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.145158485 (58h26m9.01624153s) [noise=true , address=77.90.31.10:4550 , piece_id=PDCZ7VXNHQJ6LYPG64GZRYP4MRBGVPXCWTXSRT3UV2N6HGMDE2KA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.156650047 (58h26m9.027733092s) [noise=true , address=145.79.227.15:4908 , piece_id=JIUZSMOAKBFCFAX6RQ52TCNZMYBREQCYOEM75U3H2QYR4WH3KWCA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.162020363 (58h26m9.033103398s) [noise=true , address=77.90.31.10:4539 , piece_id=27YFK55QLZDEULYABQ3OBDSPDDEIUJ2MJSD7X3A4K6HRFJMRQJUA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.198899907 (58h26m9.069982942s) [noise=true , address=77.90.31.12:4621 , piece_id=S4FVJHEPIO2MVO56ZRIJ27RMTV723MERPLJLVD2T2GAO3T7K24TA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.208468337 (58h26m9.079551382s) [noise=true , address=45.91.116.4:4945 , piece_id=6UNWQQAWWWWC2XVUCDRCTAYJDTHBXAZJMXA6KLCTOP5PYSAJCCGA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.212217697 (58h26m9.083300732s) [noise=true , address=45.91.116.12:4254 , piece_id=UCKU4TBXJC4PSYHRDPUXETXLAILDOWA5NN7RCVVNYLVUBY7FBIFA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.215832390 (58h26m9.086915435s) [noise=true , address=77.90.31.11:4579 , piece_id=JINRNEVUAP2VI5Y4CR7ALWDQIAIAFX3SP2VVZSXDDG6XPUKASE3Q, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.215862086 (58h26m9.086945161s) [seg_pos={0 0}, upload=51329]: Segment reached optimal threshold of 14 pieces. Dec 8 01:55:09.216078098 (58h26m9.087161133s) [noise=true , address=45.91.116.6:4003 , piece_id=CRONTV2B2AVZTLKU3YNIEGQZSON7XHHL7S3K6KNNJV4YR5FHQ3PA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1veqoBGw6a8AGMoigyuNJMjcEvCT93xRq2B1d2KNhs6aEmSqVF): protocol: expected piece hash; context canceled; EOF Dec 8 01:55:09.216085662 (58h26m9.087168707s) [noise=true , address=45.91.116.6:4003 , piece_id=CRONTV2B2AVZTLKU3YNIEGQZSON7XHHL7S3K6KNNJV4YR5FHQ3PA, seg_pos={0 0}, upload=51329]: Long tail cancellation due to optimal threshold Dec 8 01:55:09.216307705 (58h26m9.08739075s) [noise=true , address=145.79.227.30:4786 , piece_id=VM65RUN3GUZZFUOHFYE7ZLPWU6COCQUMY3IK7GCNWPIUZAMEBR7Q, seg_pos={0 0}, upload=51329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125TWU7R4pkmrUxsNLhgTD475aLbjUJ7psA6drJtE7oDqZXYJ4Q): protocol: expected piece hash; context canceled; EOF Dec 8 01:55:09.216313956 (58h26m9.087396991s) [noise=true , address=145.79.227.30:4786 , piece_id=VM65RUN3GUZZFUOHFYE7ZLPWU6COCQUMY3IK7GCNWPIUZAMEBR7Q, seg_pos={0 0}, upload=51329]: Long tail cancellation due to optimal threshold Dec 8 01:55:09.216414809 (58h26m9.087497844s) [noise=true , address=77.90.31.16:4760 , piece_id=Q67LYTDJCAR2TFVSAIIYCUYM7D44JFPBLCSKRNLXH26EAW5UQ3GA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1e6p1TD5CqUyWdZjkrxh2kRqMuyDiMUr9eimfeayDAKwhtBrZ8): protocol: expected piece hash; context canceled; EOF Dec 8 01:55:09.216420681 (58h26m9.087503726s) [noise=true , address=77.90.31.16:4760 , piece_id=Q67LYTDJCAR2TFVSAIIYCUYM7D44JFPBLCSKRNLXH26EAW5UQ3GA, seg_pos={0 0}, upload=51329]: Long tail cancellation due to optimal threshold Dec 8 01:55:09.216562861 (58h26m9.087645906s) [noise=true , address=45.91.116.7:4037 , piece_id=UBNFDSXQBISC5QQSNGAAZIR7LRE7FCYRVYN5MMOY3A2ALEOQBYQQ, seg_pos={0 0}, upload=51329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iNPwE2XSfXcaXAXcrH2jcyNGYeDiiWvzAcDQffrmjPwTVqVmK): protocol: expected piece hash; context canceled; EOF Dec 8 01:55:09.216570025 (58h26m9.08765307s) [noise=true , address=45.91.116.7:4037 , piece_id=UBNFDSXQBISC5QQSNGAAZIR7LRE7FCYRVYN5MMOY3A2ALEOQBYQQ, seg_pos={0 0}, upload=51329]: Long tail cancellation due to optimal threshold Dec 8 01:55:09.216585564 (58h26m9.087668609s) [seg_pos={0 0}, upload=51329]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 8 01:55:09.216654365 (58h26m9.0877374s) [upload=51329]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 01:55:09.216659064 (58h26m9.087742099s) [upload=51329]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 01:55:35.852213564 (58h26m35.723296629s) [upload=51329]: Done uploading segments... Dec 8 01:55:35.852273619 (58h26m35.723356664s) [upload=51329]: Upload finished. err: metaclient: context deadline exceeded Duration: 15.795468553s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 6.501614542s trace: https://jaeger.dev.storj.io/trace/27d92c196f473eec FAIL (2025-12-08 02:25:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 4891s) [upload=51469]: Got next segment. Inline: false Dec 8 02:04:21.143134547 (58h35m21.014217592s) [upload=51469]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 02:04:21.143141259 (58h35m21.014224304s) [upload=51469]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 02:04:22.267502706 (58h35m22.138585751s) [seg_pos={0 0}, upload=51469]: Begin upload segment... Dec 8 02:04:22.267624127 (58h35m22.138707172s) [seg_pos={0 0}, upload=51469]: Done begin upload segment. Dec 8 02:04:22.267698027 (58h35m22.138781073s) [noise=true , address=77.90.31.11:4554 , piece_id=BQP7XTRA2SBVINT4J7JAOF3ORNENBBIQN4ABMCEMQQ6FQL6QUOKA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.267712997 (58h35m22.138796042s) [noise=true , address=145.79.227.41:4087 , piece_id=XO6K3CXK7KZ5VEVXULHKQXLSSLIIWIX3VKQOSD7HY25WMNMCDLEQ, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.268434680 (58h35m22.139517715s) [noise=true , address=145.79.227.16:4910 , piece_id=ETE7ZBHT5IJUOBWCA5BGLKC2ANDEUB2M3DOMXWWS2RBWSJQBARRQ, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.268550331 (58h35m22.139633366s) [noise=true , address=145.79.227.26:4653 , piece_id=DKW72PEWPIOHPYHLI54PBU7NR3HGUBLFM3MN4XQQFWIIZ4FHIUNA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.268913111 (58h35m22.139996146s) [noise=true , address=145.79.227.23:4545 , piece_id=NBYCSU2KSHTAL223AHKHREQWKXUEQKJU46TY3SPYTKT5N5T2OUSA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.268924032 (58h35m22.140007067s) [noise=true , address=145.79.227.40:4051 , piece_id=7CQTGHFZ542WUHYX2CTOPN4PF2KDOOLDGV64THDBZCYOCXA3JWDA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.269398886 (58h35m22.140481921s) [noise=true , address=77.90.31.10:4518 , piece_id=6AR3NAFODP54SKG3SJSJSIQQFOYXO2C65WVNN4Z4XG426ZEOEDQA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.269414686 (58h35m22.140497721s) [noise=true , address=145.79.227.25:4619 , piece_id=XBNGVRIQLQUWKKBF4ULL7ZJDDT75QLY7ES2J7WHYWRD2ZLHOGU2Q, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.269753261 (58h35m22.140836296s) [noise=true , address=145.79.227.23:4547 , piece_id=5Y6V5QWLCWNOLT2GDNUEIRSG75KGNGIFLBCBGSIT7XH46RZYVDGQ, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.270016322 (58h35m22.141099367s) [noise=true , address=77.90.31.13:4628 , piece_id=LWTOCQRRA7HLI5ZPPRQ25IM22LMQNRR5H3TTOTSTGCFOYFZMABAA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.270259175 (58h35m22.14134222s) [noise=true , address=145.79.227.25:4617 , piece_id=S4ETCJ6WMCPXXQCYPPBBMHLMK2SVJV4S5VOKHGTDAGSGG6WRYH3Q, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.270655339 (58h35m22.141738374s) [noise=true , address=145.79.227.38:4977 , piece_id=73WXA33TWM4MFVTHDHW3PXJVABMCVMQYI3P42AHT3OYPKMXIVTUQ, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.271143158 (58h35m22.142246412s) [noise=true , address=145.79.227.28:4727 , piece_id=UYRK457JBF2AHCIXZ4XOIL7YHOVYZ7IPDUTB6WOZZUDEW5Q7NNCQ, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.271230565 (58h35m22.14231361s) [noise=true , address=145.79.227.39:4013 , piece_id=UI22RDXJ2DY3RFNSJVNT43XQJ6B2S3EO6SFLYHXW4VPVQJSYYF6Q, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.271662086 (58h35m22.142745121s) [noise=true , address=145.79.227.22:4511 , piece_id=GGA463DAOSZCW7SQGZW2LGFRQVNUOUJ2L2GXRXLSDPC65KVPY3TA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.272592658 (58h35m22.143675703s) [noise=true , address=145.79.227.24:4583 , piece_id=WYTXGXG77ZW4RGWX7YWDU2NH6R5CCPVBPQH3ISHUAJVKZJRONS5Q, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:24.009199123 (58h35m23.880282158s) [upload=51469]: Next returned nil segment Dec 8 02:04:24.009221235 (58h35m23.88030428s) [upload=51469]: Waiting for error group managing segments... Dec 8 02:04:24.293906896 (58h35m24.164989931s) [noise=true , address=145.79.227.24:4583 , piece_id=WYTXGXG77ZW4RGWX7YWDU2NH6R5CCPVBPQH3ISHUAJVKZJRONS5Q, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.298576738 (58h35m24.169659783s) [noise=true , address=145.79.227.28:4727 , piece_id=UYRK457JBF2AHCIXZ4XOIL7YHOVYZ7IPDUTB6WOZZUDEW5Q7NNCQ, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.301356571 (58h35m24.172439616s) [noise=true , address=145.79.227.25:4619 , piece_id=XBNGVRIQLQUWKKBF4ULL7ZJDDT75QLY7ES2J7WHYWRD2ZLHOGU2Q, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.326466765 (58h35m24.19754981s) [noise=true , address=145.79.227.16:4910 , piece_id=ETE7ZBHT5IJUOBWCA5BGLKC2ANDEUB2M3DOMXWWS2RBWSJQBARRQ, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.328633530 (58h35m24.199716586s) [noise=true , address=145.79.227.26:4653 , piece_id=DKW72PEWPIOHPYHLI54PBU7NR3HGUBLFM3MN4XQQFWIIZ4FHIUNA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.360508106 (58h35m24.231591151s) [noise=true , address=145.79.227.40:4051 , piece_id=7CQTGHFZ542WUHYX2CTOPN4PF2KDOOLDGV64THDBZCYOCXA3JWDA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.361116263 (58h35m24.232199308s) [noise=true , address=145.79.227.25:4617 , piece_id=S4ETCJ6WMCPXXQCYPPBBMHLMK2SVJV4S5VOKHGTDAGSGG6WRYH3Q, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.361695086 (58h35m24.232778131s) [noise=true , address=145.79.227.41:4087 , piece_id=XO6K3CXK7KZ5VEVXULHKQXLSSLIIWIX3VKQOSD7HY25WMNMCDLEQ, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.371520525 (58h35m24.24260356s) [noise=true , address=145.79.227.38:4977 , piece_id=73WXA33TWM4MFVTHDHW3PXJVABMCVMQYI3P42AHT3OYPKMXIVTUQ, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.381271893 (58h35m24.252354928s) [noise=true , address=145.79.227.23:4545 , piece_id=NBYCSU2KSHTAL223AHKHREQWKXUEQKJU46TY3SPYTKT5N5T2OUSA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.386403053 (58h35m24.257486098s) [noise=true , address=77.90.31.11:4554 , piece_id=BQP7XTRA2SBVINT4J7JAOF3ORNENBBIQN4ABMCEMQQ6FQL6QUOKA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.386427209 (58h35m24.257510254s) [noise=true , address=77.90.31.10:4518 , piece_id=6AR3NAFODP54SKG3SJSJSIQQFOYXO2C65WVNN4Z4XG426ZEOEDQA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.400489386 (58h35m24.271572431s) [noise=true , address=77.90.31.13:4628 , piece_id=LWTOCQRRA7HLI5ZPPRQ25IM22LMQNRR5H3TTOTSTGCFOYFZMABAA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.400513903 (58h35m24.271596938s) [seg_pos={0 0}, upload=51469]: Segment reached optimal threshold of 13 pieces. Dec 8 02:04:24.400813233 (58h35m24.271896288s) [noise=true , address=145.79.227.39:4013 , piece_id=UI22RDXJ2DY3RFNSJVNT43XQJ6B2S3EO6SFLYHXW4VPVQJSYYF6Q, seg_pos={0 0}, upload=51469]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b): protocol: write tcp 188.166.225.170:39434->145.79.227.39:4013: use of closed network connection; write tcp 188.166.225.170:39434->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39434->145.79.227.39:4013: use of closed network connection Dec 8 02:04:24.400825677 (58h35m24.271908722s) [noise=true , address=145.79.227.39:4013 , piece_id=UI22RDXJ2DY3RFNSJVNT43XQJ6B2S3EO6SFLYHXW4VPVQJSYYF6Q, seg_pos={0 0}, upload=51469]: Long tail cancellation due to optimal threshold Dec 8 02:04:24.400873027 (58h35m24.271956082s) [noise=true , address=145.79.227.22:4511 , piece_id=GGA463DAOSZCW7SQGZW2LGFRQVNUOUJ2L2GXRXLSDPC65KVPY3TA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG): protocol: write tcp 188.166.225.170:47926->145.79.227.22:4511: use of closed network connection; write tcp 188.166.225.170:47926->145.79.227.22:4511: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47926->145.79.227.22:4511: use of closed network connection Dec 8 02:04:24.400894248 (58h35m24.271977293s) [noise=true , address=145.79.227.22:4511 , piece_id=GGA463DAOSZCW7SQGZW2LGFRQVNUOUJ2L2GXRXLSDPC65KVPY3TA, seg_pos={0 0}, upload=51469]: Long tail cancellation due to optimal threshold Dec 8 02:04:24.401247129 (58h35m24.272330184s) [noise=true , address=145.79.227.23:4547 , piece_id=5Y6V5QWLCWNOLT2GDNUEIRSG75KGNGIFLBCBGSIT7XH46RZYVDGQ, seg_pos={0 0}, upload=51469]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW): protocol: read tcp 188.166.225.170:43344->145.79.227.23:4547: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 02:04:24.401260003 (58h35m24.272343059s) [noise=true , address=145.79.227.23:4547 , piece_id=5Y6V5QWLCWNOLT2GDNUEIRSG75KGNGIFLBCBGSIT7XH46RZYVDGQ, seg_pos={0 0}, upload=51469]: Long tail cancellation due to optimal threshold Dec 8 02:04:24.401285452 (58h35m24.272368497s) [seg_pos={0 0}, upload=51469]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 02:04:24.401382247 (58h35m24.272465292s) [upload=51469]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 02:04:24.401391394 (58h35m24.272474429s) [upload=51469]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 02:05:39.049922629 (58h36m38.921005704s) [upload=51469]: Done uploading segments... Dec 8 02:05:39.050003342 (58h36m38.921086377s) [upload=51469]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.501643116s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.X................X.........X...............................] duration: 5.891493817s trace: https://jaeger.dev.storj.io/trace/e7164007ac0fd55 FAIL (2025-12-08 02:25:34 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.891525827s core uplink us1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 4.990510167s trace: https://jaeger.dev.storj.io/trace/7af096edea7ffcf5 FAIL (2025-12-08 02:25:09 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.99054329s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 6.249832285s trace: https://jaeger.dev.storj.io/trace/27d92c196f473eec FAIL (2025-12-08 02:25:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 4891s) [upload=51469]: Got next segment. Inline: false Dec 8 02:04:21.143134547 (58h35m21.014217592s) [upload=51469]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 02:04:21.143141259 (58h35m21.014224304s) [upload=51469]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 02:04:22.267502706 (58h35m22.138585751s) [seg_pos={0 0}, upload=51469]: Begin upload segment... Dec 8 02:04:22.267624127 (58h35m22.138707172s) [seg_pos={0 0}, upload=51469]: Done begin upload segment. Dec 8 02:04:22.267698027 (58h35m22.138781073s) [noise=true , address=77.90.31.11:4554 , piece_id=BQP7XTRA2SBVINT4J7JAOF3ORNENBBIQN4ABMCEMQQ6FQL6QUOKA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.267712997 (58h35m22.138796042s) [noise=true , address=145.79.227.41:4087 , piece_id=XO6K3CXK7KZ5VEVXULHKQXLSSLIIWIX3VKQOSD7HY25WMNMCDLEQ, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.268434680 (58h35m22.139517715s) [noise=true , address=145.79.227.16:4910 , piece_id=ETE7ZBHT5IJUOBWCA5BGLKC2ANDEUB2M3DOMXWWS2RBWSJQBARRQ, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.268550331 (58h35m22.139633366s) [noise=true , address=145.79.227.26:4653 , piece_id=DKW72PEWPIOHPYHLI54PBU7NR3HGUBLFM3MN4XQQFWIIZ4FHIUNA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.268913111 (58h35m22.139996146s) [noise=true , address=145.79.227.23:4545 , piece_id=NBYCSU2KSHTAL223AHKHREQWKXUEQKJU46TY3SPYTKT5N5T2OUSA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.268924032 (58h35m22.140007067s) [noise=true , address=145.79.227.40:4051 , piece_id=7CQTGHFZ542WUHYX2CTOPN4PF2KDOOLDGV64THDBZCYOCXA3JWDA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.269398886 (58h35m22.140481921s) [noise=true , address=77.90.31.10:4518 , piece_id=6AR3NAFODP54SKG3SJSJSIQQFOYXO2C65WVNN4Z4XG426ZEOEDQA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.269414686 (58h35m22.140497721s) [noise=true , address=145.79.227.25:4619 , piece_id=XBNGVRIQLQUWKKBF4ULL7ZJDDT75QLY7ES2J7WHYWRD2ZLHOGU2Q, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.269753261 (58h35m22.140836296s) [noise=true , address=145.79.227.23:4547 , piece_id=5Y6V5QWLCWNOLT2GDNUEIRSG75KGNGIFLBCBGSIT7XH46RZYVDGQ, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.270016322 (58h35m22.141099367s) [noise=true , address=77.90.31.13:4628 , piece_id=LWTOCQRRA7HLI5ZPPRQ25IM22LMQNRR5H3TTOTSTGCFOYFZMABAA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.270259175 (58h35m22.14134222s) [noise=true , address=145.79.227.25:4617 , piece_id=S4ETCJ6WMCPXXQCYPPBBMHLMK2SVJV4S5VOKHGTDAGSGG6WRYH3Q, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.270655339 (58h35m22.141738374s) [noise=true , address=145.79.227.38:4977 , piece_id=73WXA33TWM4MFVTHDHW3PXJVABMCVMQYI3P42AHT3OYPKMXIVTUQ, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.271143158 (58h35m22.142246412s) [noise=true , address=145.79.227.28:4727 , piece_id=UYRK457JBF2AHCIXZ4XOIL7YHOVYZ7IPDUTB6WOZZUDEW5Q7NNCQ, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.271230565 (58h35m22.14231361s) [noise=true , address=145.79.227.39:4013 , piece_id=UI22RDXJ2DY3RFNSJVNT43XQJ6B2S3EO6SFLYHXW4VPVQJSYYF6Q, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.271662086 (58h35m22.142745121s) [noise=true , address=145.79.227.22:4511 , piece_id=GGA463DAOSZCW7SQGZW2LGFRQVNUOUJ2L2GXRXLSDPC65KVPY3TA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.272592658 (58h35m22.143675703s) [noise=true , address=145.79.227.24:4583 , piece_id=WYTXGXG77ZW4RGWX7YWDU2NH6R5CCPVBPQH3ISHUAJVKZJRONS5Q, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:24.009199123 (58h35m23.880282158s) [upload=51469]: Next returned nil segment Dec 8 02:04:24.009221235 (58h35m23.88030428s) [upload=51469]: Waiting for error group managing segments... Dec 8 02:04:24.293906896 (58h35m24.164989931s) [noise=true , address=145.79.227.24:4583 , piece_id=WYTXGXG77ZW4RGWX7YWDU2NH6R5CCPVBPQH3ISHUAJVKZJRONS5Q, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.298576738 (58h35m24.169659783s) [noise=true , address=145.79.227.28:4727 , piece_id=UYRK457JBF2AHCIXZ4XOIL7YHOVYZ7IPDUTB6WOZZUDEW5Q7NNCQ, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.301356571 (58h35m24.172439616s) [noise=true , address=145.79.227.25:4619 , piece_id=XBNGVRIQLQUWKKBF4ULL7ZJDDT75QLY7ES2J7WHYWRD2ZLHOGU2Q, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.326466765 (58h35m24.19754981s) [noise=true , address=145.79.227.16:4910 , piece_id=ETE7ZBHT5IJUOBWCA5BGLKC2ANDEUB2M3DOMXWWS2RBWSJQBARRQ, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.328633530 (58h35m24.199716586s) [noise=true , address=145.79.227.26:4653 , piece_id=DKW72PEWPIOHPYHLI54PBU7NR3HGUBLFM3MN4XQQFWIIZ4FHIUNA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.360508106 (58h35m24.231591151s) [noise=true , address=145.79.227.40:4051 , piece_id=7CQTGHFZ542WUHYX2CTOPN4PF2KDOOLDGV64THDBZCYOCXA3JWDA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.361116263 (58h35m24.232199308s) [noise=true , address=145.79.227.25:4617 , piece_id=S4ETCJ6WMCPXXQCYPPBBMHLMK2SVJV4S5VOKHGTDAGSGG6WRYH3Q, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.361695086 (58h35m24.232778131s) [noise=true , address=145.79.227.41:4087 , piece_id=XO6K3CXK7KZ5VEVXULHKQXLSSLIIWIX3VKQOSD7HY25WMNMCDLEQ, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.371520525 (58h35m24.24260356s) [noise=true , address=145.79.227.38:4977 , piece_id=73WXA33TWM4MFVTHDHW3PXJVABMCVMQYI3P42AHT3OYPKMXIVTUQ, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.381271893 (58h35m24.252354928s) [noise=true , address=145.79.227.23:4545 , piece_id=NBYCSU2KSHTAL223AHKHREQWKXUEQKJU46TY3SPYTKT5N5T2OUSA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.386403053 (58h35m24.257486098s) [noise=true , address=77.90.31.11:4554 , piece_id=BQP7XTRA2SBVINT4J7JAOF3ORNENBBIQN4ABMCEMQQ6FQL6QUOKA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.386427209 (58h35m24.257510254s) [noise=true , address=77.90.31.10:4518 , piece_id=6AR3NAFODP54SKG3SJSJSIQQFOYXO2C65WVNN4Z4XG426ZEOEDQA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.400489386 (58h35m24.271572431s) [noise=true , address=77.90.31.13:4628 , piece_id=LWTOCQRRA7HLI5ZPPRQ25IM22LMQNRR5H3TTOTSTGCFOYFZMABAA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.400513903 (58h35m24.271596938s) [seg_pos={0 0}, upload=51469]: Segment reached optimal threshold of 13 pieces. Dec 8 02:04:24.400813233 (58h35m24.271896288s) [noise=true , address=145.79.227.39:4013 , piece_id=UI22RDXJ2DY3RFNSJVNT43XQJ6B2S3EO6SFLYHXW4VPVQJSYYF6Q, seg_pos={0 0}, upload=51469]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b): protocol: write tcp 188.166.225.170:39434->145.79.227.39:4013: use of closed network connection; write tcp 188.166.225.170:39434->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39434->145.79.227.39:4013: use of closed network connection Dec 8 02:04:24.400825677 (58h35m24.271908722s) [noise=true , address=145.79.227.39:4013 , piece_id=UI22RDXJ2DY3RFNSJVNT43XQJ6B2S3EO6SFLYHXW4VPVQJSYYF6Q, seg_pos={0 0}, upload=51469]: Long tail cancellation due to optimal threshold Dec 8 02:04:24.400873027 (58h35m24.271956082s) [noise=true , address=145.79.227.22:4511 , piece_id=GGA463DAOSZCW7SQGZW2LGFRQVNUOUJ2L2GXRXLSDPC65KVPY3TA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG): protocol: write tcp 188.166.225.170:47926->145.79.227.22:4511: use of closed network connection; write tcp 188.166.225.170:47926->145.79.227.22:4511: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47926->145.79.227.22:4511: use of closed network connection Dec 8 02:04:24.400894248 (58h35m24.271977293s) [noise=true , address=145.79.227.22:4511 , piece_id=GGA463DAOSZCW7SQGZW2LGFRQVNUOUJ2L2GXRXLSDPC65KVPY3TA, seg_pos={0 0}, upload=51469]: Long tail cancellation due to optimal threshold Dec 8 02:04:24.401247129 (58h35m24.272330184s) [noise=true , address=145.79.227.23:4547 , piece_id=5Y6V5QWLCWNOLT2GDNUEIRSG75KGNGIFLBCBGSIT7XH46RZYVDGQ, seg_pos={0 0}, upload=51469]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW): protocol: read tcp 188.166.225.170:43344->145.79.227.23:4547: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 02:04:24.401260003 (58h35m24.272343059s) [noise=true , address=145.79.227.23:4547 , piece_id=5Y6V5QWLCWNOLT2GDNUEIRSG75KGNGIFLBCBGSIT7XH46RZYVDGQ, seg_pos={0 0}, upload=51469]: Long tail cancellation due to optimal threshold Dec 8 02:04:24.401285452 (58h35m24.272368497s) [seg_pos={0 0}, upload=51469]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 02:04:24.401382247 (58h35m24.272465292s) [upload=51469]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 02:04:24.401391394 (58h35m24.272474429s) [upload=51469]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 02:05:39.049922629 (58h36m38.921005704s) [upload=51469]: Done uploading segments... Dec 8 02:05:39.050003342 (58h36m38.921086377s) [upload=51469]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.249865338s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [......................X......X................X....X........] duration: 3.452591744s trace: https://jaeger.dev.storj.io/trace/491130a80ad889c4 FAIL (2025-12-08 02:25:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.452671315s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 5.965871475s trace: https://jaeger.dev.storj.io/trace/57258282a8e974fd FAIL (2025-12-08 02:24:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== UA4TAJ2FSYO5S4NXECVMMMCXKFL5CAHPQBQJ6WREQ3YFC3Q, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.202906305 (58h26m7.07398934s) [noise=true , address=45.91.116.4:4945 , piece_id=6UNWQQAWWWWC2XVUCDRCTAYJDTHBXAZJMXA6KLCTOP5PYSAJCCGA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.203371201 (58h26m7.074454246s) [noise=true , address=145.79.227.11:4740 , piece_id=KL54LL2UQRT6MH2K3NQQJB3BKPX4QRWQMXRK27MW2U7CHFDCCWFA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.203861354 (58h26m7.07494439s) [noise=true , address=77.90.31.12:4621 , piece_id=S4FVJHEPIO2MVO56ZRIJ27RMTV723MERPLJLVD2T2GAO3T7K24TA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.204290401 (58h26m7.075373446s) [noise=true , address=145.79.227.34:4928 , piece_id=DRKF2DRSCBAYSZO66A2ZON2JS6HIJIT2YUWFZRS3W2CKFTV3NLVA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.204711363 (58h26m7.075794398s) [noise=true , address=145.79.227.15:4908 , piece_id=JIUZSMOAKBFCFAX6RQ52TCNZMYBREQCYOEM75U3H2QYR4WH3KWCA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.205118208 (58h26m7.076201253s) [noise=true , address=77.90.31.18:4825 , piece_id=EJ4LO4TCEFB4OQQLFOEUYGQKK462ZPXXEIIY5CHF2PERXWDGQNDA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.205554277 (58h26m7.076637323s) [noise=true , address=145.79.227.30:4786 , piece_id=VM65RUN3GUZZFUOHFYE7ZLPWU6COCQUMY3IK7GCNWPIUZAMEBR7Q, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.206116137 (58h26m7.077199182s) [noise=true , address=145.79.227.25:4627 , piece_id=K7UYU3W2CACUAQUUL72GG5F7DP5CHAHZ4M47TZG6NBN5Z6ZCDRBA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.206664562 (58h26m7.077747607s) [noise=true , address=77.90.31.16:4760 , piece_id=Q67LYTDJCAR2TFVSAIIYCUYM7D44JFPBLCSKRNLXH26EAW5UQ3GA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.207102556 (58h26m7.078185601s) [noise=true , address=45.91.116.7:4037 , piece_id=UBNFDSXQBISC5QQSNGAAZIR7LRE7FCYRVYN5MMOY3A2ALEOQBYQQ, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.207519911 (58h26m7.078602956s) [noise=true , address=77.90.31.11:4579 , piece_id=JINRNEVUAP2VI5Y4CR7ALWDQIAIAFX3SP2VVZSXDDG6XPUKASE3Q, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.207967393 (58h26m7.079050438s) [noise=true , address=77.90.31.10:4550 , piece_id=PDCZ7VXNHQJ6LYPG64GZRYP4MRBGVPXCWTXSRT3UV2N6HGMDE2KA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.208416427 (58h26m7.079499472s) [noise=true , address=45.91.116.6:4003 , piece_id=CRONTV2B2AVZTLKU3YNIEGQZSON7XHHL7S3K6KNNJV4YR5FHQ3PA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.208830746 (58h26m7.079913791s) [noise=true , address=45.91.116.12:4254 , piece_id=UCKU4TBXJC4PSYHRDPUXETXLAILDOWA5NN7RCVVNYLVUBY7FBIFA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.209255245 (58h26m7.080338291s) [noise=true , address=145.79.227.16:4933 , piece_id=LHF64L3SWXFVZ5X5D67I44WSJNKRFB5G4YP7DFHDWOAYE23II3OA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.209662921 (58h26m7.080745966s) [noise=true , address=145.79.227.18:4010 , piece_id=S2EOTEV4TBNNDOBN7TIX76AJE2WI5P5DM3JK2NJB25TISYLS7VSQ, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.214493680 (58h26m7.085576725s) [noise=true , address=77.90.31.10:4539 , piece_id=27YFK55QLZDEULYABQ3OBDSPDDEIUJ2MJSD7X3A4K6HRFJMRQJUA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:08.555121654 (58h26m8.426204699s) [upload=51329]: Next returned nil segment Dec 8 01:55:08.555145519 (58h26m8.426228564s) [upload=51329]: Waiting for error group managing segments... Dec 8 01:55:08.935193756 (58h26m8.806276791s) [noise=true , address=64.49.13.27:4957 , piece_id=7METNUA4TAJ2FSYO5S4NXECVMMMCXKFL5CAHPQBQJ6WREQ3YFC3Q, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.093051314 (58h26m8.964134359s) [noise=true , address=145.79.227.16:4933 , piece_id=LHF64L3SWXFVZ5X5D67I44WSJNKRFB5G4YP7DFHDWOAYE23II3OA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.098812273 (58h26m8.969895318s) [noise=true , address=145.79.227.18:4010 , piece_id=S2EOTEV4TBNNDOBN7TIX76AJE2WI5P5DM3JK2NJB25TISYLS7VSQ, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.102616718 (58h26m8.973699763s) [noise=true , address=145.79.227.25:4627 , piece_id=K7UYU3W2CACUAQUUL72GG5F7DP5CHAHZ4M47TZG6NBN5Z6ZCDRBA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.118902949 (58h26m8.989985994s) [noise=true , address=145.79.227.34:4928 , piece_id=DRKF2DRSCBAYSZO66A2ZON2JS6HIJIT2YUWFZRS3W2CKFTV3NLVA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.124783838 (58h26m8.995866883s) [noise=true , address=77.90.31.18:4825 , piece_id=EJ4LO4TCEFB4OQQLFOEUYGQKK462ZPXXEIIY5CHF2PERXWDGQNDA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.126718282 (58h26m8.997801327s) [noise=true , address=145.79.227.11:4740 , piece_id=KL54LL2UQRT6MH2K3NQQJB3BKPX4QRWQMXRK27MW2U7CHFDCCWFA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.145158485 (58h26m9.01624153s) [noise=true , address=77.90.31.10:4550 , piece_id=PDCZ7VXNHQJ6LYPG64GZRYP4MRBGVPXCWTXSRT3UV2N6HGMDE2KA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.156650047 (58h26m9.027733092s) [noise=true , address=145.79.227.15:4908 , piece_id=JIUZSMOAKBFCFAX6RQ52TCNZMYBREQCYOEM75U3H2QYR4WH3KWCA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.162020363 (58h26m9.033103398s) [noise=true , address=77.90.31.10:4539 , piece_id=27YFK55QLZDEULYABQ3OBDSPDDEIUJ2MJSD7X3A4K6HRFJMRQJUA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.198899907 (58h26m9.069982942s) [noise=true , address=77.90.31.12:4621 , piece_id=S4FVJHEPIO2MVO56ZRIJ27RMTV723MERPLJLVD2T2GAO3T7K24TA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.208468337 (58h26m9.079551382s) [noise=true , address=45.91.116.4:4945 , piece_id=6UNWQQAWWWWC2XVUCDRCTAYJDTHBXAZJMXA6KLCTOP5PYSAJCCGA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.212217697 (58h26m9.083300732s) [noise=true , address=45.91.116.12:4254 , piece_id=UCKU4TBXJC4PSYHRDPUXETXLAILDOWA5NN7RCVVNYLVUBY7FBIFA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.215832390 (58h26m9.086915435s) [noise=true , address=77.90.31.11:4579 , piece_id=JINRNEVUAP2VI5Y4CR7ALWDQIAIAFX3SP2VVZSXDDG6XPUKASE3Q, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.215862086 (58h26m9.086945161s) [seg_pos={0 0}, upload=51329]: Segment reached optimal threshold of 14 pieces. Dec 8 01:55:09.216078098 (58h26m9.087161133s) [noise=true , address=45.91.116.6:4003 , piece_id=CRONTV2B2AVZTLKU3YNIEGQZSON7XHHL7S3K6KNNJV4YR5FHQ3PA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1veqoBGw6a8AGMoigyuNJMjcEvCT93xRq2B1d2KNhs6aEmSqVF): protocol: expected piece hash; context canceled; EOF Dec 8 01:55:09.216085662 (58h26m9.087168707s) [noise=true , address=45.91.116.6:4003 , piece_id=CRONTV2B2AVZTLKU3YNIEGQZSON7XHHL7S3K6KNNJV4YR5FHQ3PA, seg_pos={0 0}, upload=51329]: Long tail cancellation due to optimal threshold Dec 8 01:55:09.216307705 (58h26m9.08739075s) [noise=true , address=145.79.227.30:4786 , piece_id=VM65RUN3GUZZFUOHFYE7ZLPWU6COCQUMY3IK7GCNWPIUZAMEBR7Q, seg_pos={0 0}, upload=51329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125TWU7R4pkmrUxsNLhgTD475aLbjUJ7psA6drJtE7oDqZXYJ4Q): protocol: expected piece hash; context canceled; EOF Dec 8 01:55:09.216313956 (58h26m9.087396991s) [noise=true , address=145.79.227.30:4786 , piece_id=VM65RUN3GUZZFUOHFYE7ZLPWU6COCQUMY3IK7GCNWPIUZAMEBR7Q, seg_pos={0 0}, upload=51329]: Long tail cancellation due to optimal threshold Dec 8 01:55:09.216414809 (58h26m9.087497844s) [noise=true , address=77.90.31.16:4760 , piece_id=Q67LYTDJCAR2TFVSAIIYCUYM7D44JFPBLCSKRNLXH26EAW5UQ3GA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1e6p1TD5CqUyWdZjkrxh2kRqMuyDiMUr9eimfeayDAKwhtBrZ8): protocol: expected piece hash; context canceled; EOF Dec 8 01:55:09.216420681 (58h26m9.087503726s) [noise=true , address=77.90.31.16:4760 , piece_id=Q67LYTDJCAR2TFVSAIIYCUYM7D44JFPBLCSKRNLXH26EAW5UQ3GA, seg_pos={0 0}, upload=51329]: Long tail cancellation due to optimal threshold Dec 8 01:55:09.216562861 (58h26m9.087645906s) [noise=true , address=45.91.116.7:4037 , piece_id=UBNFDSXQBISC5QQSNGAAZIR7LRE7FCYRVYN5MMOY3A2ALEOQBYQQ, seg_pos={0 0}, upload=51329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iNPwE2XSfXcaXAXcrH2jcyNGYeDiiWvzAcDQffrmjPwTVqVmK): protocol: expected piece hash; context canceled; EOF Dec 8 01:55:09.216570025 (58h26m9.08765307s) [noise=true , address=45.91.116.7:4037 , piece_id=UBNFDSXQBISC5QQSNGAAZIR7LRE7FCYRVYN5MMOY3A2ALEOQBYQQ, seg_pos={0 0}, upload=51329]: Long tail cancellation due to optimal threshold Dec 8 01:55:09.216585564 (58h26m9.087668609s) [seg_pos={0 0}, upload=51329]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 8 01:55:09.216654365 (58h26m9.0877374s) [upload=51329]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 01:55:09.216659064 (58h26m9.087742099s) [upload=51329]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 01:55:35.852213564 (58h26m35.723296629s) [upload=51329]: Done uploading segments... Dec 8 01:55:35.852273619 (58h26m35.723356664s) [upload=51329]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.965890922s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..X................X.........X..............................] duration: 7.20294605s trace: https://jaeger.dev.storj.io/trace/e7164007ac0fd55 FAIL (2025-12-08 02:24:53 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 7.202973722s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 5.819422661s trace: https://jaeger.dev.storj.io/trace/4030003299248e86 FAIL (2025-12-08 02:24:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.819450583s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 4.945691154s trace: https://jaeger.dev.storj.io/trace/7af096edea7ffcf5 FAIL (2025-12-08 02:24:23 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.945715461s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 5.952566449s trace: https://jaeger.dev.storj.io/trace/4030003299248e86 FAIL (2025-12-08 02:24:23 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.952596226s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 6.400599663s trace: https://jaeger.dev.storj.io/trace/57258282a8e974fd FAIL (2025-12-08 02:24:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== UA4TAJ2FSYO5S4NXECVMMMCXKFL5CAHPQBQJ6WREQ3YFC3Q, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.202906305 (58h26m7.07398934s) [noise=true , address=45.91.116.4:4945 , piece_id=6UNWQQAWWWWC2XVUCDRCTAYJDTHBXAZJMXA6KLCTOP5PYSAJCCGA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.203371201 (58h26m7.074454246s) [noise=true , address=145.79.227.11:4740 , piece_id=KL54LL2UQRT6MH2K3NQQJB3BKPX4QRWQMXRK27MW2U7CHFDCCWFA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.203861354 (58h26m7.07494439s) [noise=true , address=77.90.31.12:4621 , piece_id=S4FVJHEPIO2MVO56ZRIJ27RMTV723MERPLJLVD2T2GAO3T7K24TA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.204290401 (58h26m7.075373446s) [noise=true , address=145.79.227.34:4928 , piece_id=DRKF2DRSCBAYSZO66A2ZON2JS6HIJIT2YUWFZRS3W2CKFTV3NLVA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.204711363 (58h26m7.075794398s) [noise=true , address=145.79.227.15:4908 , piece_id=JIUZSMOAKBFCFAX6RQ52TCNZMYBREQCYOEM75U3H2QYR4WH3KWCA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.205118208 (58h26m7.076201253s) [noise=true , address=77.90.31.18:4825 , piece_id=EJ4LO4TCEFB4OQQLFOEUYGQKK462ZPXXEIIY5CHF2PERXWDGQNDA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.205554277 (58h26m7.076637323s) [noise=true , address=145.79.227.30:4786 , piece_id=VM65RUN3GUZZFUOHFYE7ZLPWU6COCQUMY3IK7GCNWPIUZAMEBR7Q, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.206116137 (58h26m7.077199182s) [noise=true , address=145.79.227.25:4627 , piece_id=K7UYU3W2CACUAQUUL72GG5F7DP5CHAHZ4M47TZG6NBN5Z6ZCDRBA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.206664562 (58h26m7.077747607s) [noise=true , address=77.90.31.16:4760 , piece_id=Q67LYTDJCAR2TFVSAIIYCUYM7D44JFPBLCSKRNLXH26EAW5UQ3GA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.207102556 (58h26m7.078185601s) [noise=true , address=45.91.116.7:4037 , piece_id=UBNFDSXQBISC5QQSNGAAZIR7LRE7FCYRVYN5MMOY3A2ALEOQBYQQ, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.207519911 (58h26m7.078602956s) [noise=true , address=77.90.31.11:4579 , piece_id=JINRNEVUAP2VI5Y4CR7ALWDQIAIAFX3SP2VVZSXDDG6XPUKASE3Q, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.207967393 (58h26m7.079050438s) [noise=true , address=77.90.31.10:4550 , piece_id=PDCZ7VXNHQJ6LYPG64GZRYP4MRBGVPXCWTXSRT3UV2N6HGMDE2KA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.208416427 (58h26m7.079499472s) [noise=true , address=45.91.116.6:4003 , piece_id=CRONTV2B2AVZTLKU3YNIEGQZSON7XHHL7S3K6KNNJV4YR5FHQ3PA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.208830746 (58h26m7.079913791s) [noise=true , address=45.91.116.12:4254 , piece_id=UCKU4TBXJC4PSYHRDPUXETXLAILDOWA5NN7RCVVNYLVUBY7FBIFA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.209255245 (58h26m7.080338291s) [noise=true , address=145.79.227.16:4933 , piece_id=LHF64L3SWXFVZ5X5D67I44WSJNKRFB5G4YP7DFHDWOAYE23II3OA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.209662921 (58h26m7.080745966s) [noise=true , address=145.79.227.18:4010 , piece_id=S2EOTEV4TBNNDOBN7TIX76AJE2WI5P5DM3JK2NJB25TISYLS7VSQ, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.214493680 (58h26m7.085576725s) [noise=true , address=77.90.31.10:4539 , piece_id=27YFK55QLZDEULYABQ3OBDSPDDEIUJ2MJSD7X3A4K6HRFJMRQJUA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:08.555121654 (58h26m8.426204699s) [upload=51329]: Next returned nil segment Dec 8 01:55:08.555145519 (58h26m8.426228564s) [upload=51329]: Waiting for error group managing segments... Dec 8 01:55:08.935193756 (58h26m8.806276791s) [noise=true , address=64.49.13.27:4957 , piece_id=7METNUA4TAJ2FSYO5S4NXECVMMMCXKFL5CAHPQBQJ6WREQ3YFC3Q, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.093051314 (58h26m8.964134359s) [noise=true , address=145.79.227.16:4933 , piece_id=LHF64L3SWXFVZ5X5D67I44WSJNKRFB5G4YP7DFHDWOAYE23II3OA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.098812273 (58h26m8.969895318s) [noise=true , address=145.79.227.18:4010 , piece_id=S2EOTEV4TBNNDOBN7TIX76AJE2WI5P5DM3JK2NJB25TISYLS7VSQ, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.102616718 (58h26m8.973699763s) [noise=true , address=145.79.227.25:4627 , piece_id=K7UYU3W2CACUAQUUL72GG5F7DP5CHAHZ4M47TZG6NBN5Z6ZCDRBA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.118902949 (58h26m8.989985994s) [noise=true , address=145.79.227.34:4928 , piece_id=DRKF2DRSCBAYSZO66A2ZON2JS6HIJIT2YUWFZRS3W2CKFTV3NLVA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.124783838 (58h26m8.995866883s) [noise=true , address=77.90.31.18:4825 , piece_id=EJ4LO4TCEFB4OQQLFOEUYGQKK462ZPXXEIIY5CHF2PERXWDGQNDA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.126718282 (58h26m8.997801327s) [noise=true , address=145.79.227.11:4740 , piece_id=KL54LL2UQRT6MH2K3NQQJB3BKPX4QRWQMXRK27MW2U7CHFDCCWFA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.145158485 (58h26m9.01624153s) [noise=true , address=77.90.31.10:4550 , piece_id=PDCZ7VXNHQJ6LYPG64GZRYP4MRBGVPXCWTXSRT3UV2N6HGMDE2KA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.156650047 (58h26m9.027733092s) [noise=true , address=145.79.227.15:4908 , piece_id=JIUZSMOAKBFCFAX6RQ52TCNZMYBREQCYOEM75U3H2QYR4WH3KWCA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.162020363 (58h26m9.033103398s) [noise=true , address=77.90.31.10:4539 , piece_id=27YFK55QLZDEULYABQ3OBDSPDDEIUJ2MJSD7X3A4K6HRFJMRQJUA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.198899907 (58h26m9.069982942s) [noise=true , address=77.90.31.12:4621 , piece_id=S4FVJHEPIO2MVO56ZRIJ27RMTV723MERPLJLVD2T2GAO3T7K24TA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.208468337 (58h26m9.079551382s) [noise=true , address=45.91.116.4:4945 , piece_id=6UNWQQAWWWWC2XVUCDRCTAYJDTHBXAZJMXA6KLCTOP5PYSAJCCGA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.212217697 (58h26m9.083300732s) [noise=true , address=45.91.116.12:4254 , piece_id=UCKU4TBXJC4PSYHRDPUXETXLAILDOWA5NN7RCVVNYLVUBY7FBIFA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.215832390 (58h26m9.086915435s) [noise=true , address=77.90.31.11:4579 , piece_id=JINRNEVUAP2VI5Y4CR7ALWDQIAIAFX3SP2VVZSXDDG6XPUKASE3Q, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.215862086 (58h26m9.086945161s) [seg_pos={0 0}, upload=51329]: Segment reached optimal threshold of 14 pieces. Dec 8 01:55:09.216078098 (58h26m9.087161133s) [noise=true , address=45.91.116.6:4003 , piece_id=CRONTV2B2AVZTLKU3YNIEGQZSON7XHHL7S3K6KNNJV4YR5FHQ3PA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1veqoBGw6a8AGMoigyuNJMjcEvCT93xRq2B1d2KNhs6aEmSqVF): protocol: expected piece hash; context canceled; EOF Dec 8 01:55:09.216085662 (58h26m9.087168707s) [noise=true , address=45.91.116.6:4003 , piece_id=CRONTV2B2AVZTLKU3YNIEGQZSON7XHHL7S3K6KNNJV4YR5FHQ3PA, seg_pos={0 0}, upload=51329]: Long tail cancellation due to optimal threshold Dec 8 01:55:09.216307705 (58h26m9.08739075s) [noise=true , address=145.79.227.30:4786 , piece_id=VM65RUN3GUZZFUOHFYE7ZLPWU6COCQUMY3IK7GCNWPIUZAMEBR7Q, seg_pos={0 0}, upload=51329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125TWU7R4pkmrUxsNLhgTD475aLbjUJ7psA6drJtE7oDqZXYJ4Q): protocol: expected piece hash; context canceled; EOF Dec 8 01:55:09.216313956 (58h26m9.087396991s) [noise=true , address=145.79.227.30:4786 , piece_id=VM65RUN3GUZZFUOHFYE7ZLPWU6COCQUMY3IK7GCNWPIUZAMEBR7Q, seg_pos={0 0}, upload=51329]: Long tail cancellation due to optimal threshold Dec 8 01:55:09.216414809 (58h26m9.087497844s) [noise=true , address=77.90.31.16:4760 , piece_id=Q67LYTDJCAR2TFVSAIIYCUYM7D44JFPBLCSKRNLXH26EAW5UQ3GA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1e6p1TD5CqUyWdZjkrxh2kRqMuyDiMUr9eimfeayDAKwhtBrZ8): protocol: expected piece hash; context canceled; EOF Dec 8 01:55:09.216420681 (58h26m9.087503726s) [noise=true , address=77.90.31.16:4760 , piece_id=Q67LYTDJCAR2TFVSAIIYCUYM7D44JFPBLCSKRNLXH26EAW5UQ3GA, seg_pos={0 0}, upload=51329]: Long tail cancellation due to optimal threshold Dec 8 01:55:09.216562861 (58h26m9.087645906s) [noise=true , address=45.91.116.7:4037 , piece_id=UBNFDSXQBISC5QQSNGAAZIR7LRE7FCYRVYN5MMOY3A2ALEOQBYQQ, seg_pos={0 0}, upload=51329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iNPwE2XSfXcaXAXcrH2jcyNGYeDiiWvzAcDQffrmjPwTVqVmK): protocol: expected piece hash; context canceled; EOF Dec 8 01:55:09.216570025 (58h26m9.08765307s) [noise=true , address=45.91.116.7:4037 , piece_id=UBNFDSXQBISC5QQSNGAAZIR7LRE7FCYRVYN5MMOY3A2ALEOQBYQQ, seg_pos={0 0}, upload=51329]: Long tail cancellation due to optimal threshold Dec 8 01:55:09.216585564 (58h26m9.087668609s) [seg_pos={0 0}, upload=51329]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 8 01:55:09.216654365 (58h26m9.0877374s) [upload=51329]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 01:55:09.216659064 (58h26m9.087742099s) [upload=51329]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 01:55:35.852213564 (58h26m35.723296629s) [upload=51329]: Done uploading segments... Dec 8 01:55:35.852273619 (58h26m35.723356664s) [upload=51329]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.400664847s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...X................X.........X.............................] duration: 6.043858546s trace: https://jaeger.dev.storj.io/trace/e7164007ac0fd55 FAIL (2025-12-08 02:24:03 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 6.04389674s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 7.445393681s trace: https://jaeger.dev.storj.io/trace/27d92c196f473eec FAIL (2025-12-08 02:23:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 4891s) [upload=51469]: Got next segment. Inline: false Dec 8 02:04:21.143134547 (58h35m21.014217592s) [upload=51469]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 02:04:21.143141259 (58h35m21.014224304s) [upload=51469]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 02:04:22.267502706 (58h35m22.138585751s) [seg_pos={0 0}, upload=51469]: Begin upload segment... Dec 8 02:04:22.267624127 (58h35m22.138707172s) [seg_pos={0 0}, upload=51469]: Done begin upload segment. Dec 8 02:04:22.267698027 (58h35m22.138781073s) [noise=true , address=77.90.31.11:4554 , piece_id=BQP7XTRA2SBVINT4J7JAOF3ORNENBBIQN4ABMCEMQQ6FQL6QUOKA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.267712997 (58h35m22.138796042s) [noise=true , address=145.79.227.41:4087 , piece_id=XO6K3CXK7KZ5VEVXULHKQXLSSLIIWIX3VKQOSD7HY25WMNMCDLEQ, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.268434680 (58h35m22.139517715s) [noise=true , address=145.79.227.16:4910 , piece_id=ETE7ZBHT5IJUOBWCA5BGLKC2ANDEUB2M3DOMXWWS2RBWSJQBARRQ, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.268550331 (58h35m22.139633366s) [noise=true , address=145.79.227.26:4653 , piece_id=DKW72PEWPIOHPYHLI54PBU7NR3HGUBLFM3MN4XQQFWIIZ4FHIUNA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.268913111 (58h35m22.139996146s) [noise=true , address=145.79.227.23:4545 , piece_id=NBYCSU2KSHTAL223AHKHREQWKXUEQKJU46TY3SPYTKT5N5T2OUSA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.268924032 (58h35m22.140007067s) [noise=true , address=145.79.227.40:4051 , piece_id=7CQTGHFZ542WUHYX2CTOPN4PF2KDOOLDGV64THDBZCYOCXA3JWDA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.269398886 (58h35m22.140481921s) [noise=true , address=77.90.31.10:4518 , piece_id=6AR3NAFODP54SKG3SJSJSIQQFOYXO2C65WVNN4Z4XG426ZEOEDQA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.269414686 (58h35m22.140497721s) [noise=true , address=145.79.227.25:4619 , piece_id=XBNGVRIQLQUWKKBF4ULL7ZJDDT75QLY7ES2J7WHYWRD2ZLHOGU2Q, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.269753261 (58h35m22.140836296s) [noise=true , address=145.79.227.23:4547 , piece_id=5Y6V5QWLCWNOLT2GDNUEIRSG75KGNGIFLBCBGSIT7XH46RZYVDGQ, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.270016322 (58h35m22.141099367s) [noise=true , address=77.90.31.13:4628 , piece_id=LWTOCQRRA7HLI5ZPPRQ25IM22LMQNRR5H3TTOTSTGCFOYFZMABAA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.270259175 (58h35m22.14134222s) [noise=true , address=145.79.227.25:4617 , piece_id=S4ETCJ6WMCPXXQCYPPBBMHLMK2SVJV4S5VOKHGTDAGSGG6WRYH3Q, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.270655339 (58h35m22.141738374s) [noise=true , address=145.79.227.38:4977 , piece_id=73WXA33TWM4MFVTHDHW3PXJVABMCVMQYI3P42AHT3OYPKMXIVTUQ, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.271143158 (58h35m22.142246412s) [noise=true , address=145.79.227.28:4727 , piece_id=UYRK457JBF2AHCIXZ4XOIL7YHOVYZ7IPDUTB6WOZZUDEW5Q7NNCQ, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.271230565 (58h35m22.14231361s) [noise=true , address=145.79.227.39:4013 , piece_id=UI22RDXJ2DY3RFNSJVNT43XQJ6B2S3EO6SFLYHXW4VPVQJSYYF6Q, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.271662086 (58h35m22.142745121s) [noise=true , address=145.79.227.22:4511 , piece_id=GGA463DAOSZCW7SQGZW2LGFRQVNUOUJ2L2GXRXLSDPC65KVPY3TA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.272592658 (58h35m22.143675703s) [noise=true , address=145.79.227.24:4583 , piece_id=WYTXGXG77ZW4RGWX7YWDU2NH6R5CCPVBPQH3ISHUAJVKZJRONS5Q, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:24.009199123 (58h35m23.880282158s) [upload=51469]: Next returned nil segment Dec 8 02:04:24.009221235 (58h35m23.88030428s) [upload=51469]: Waiting for error group managing segments... Dec 8 02:04:24.293906896 (58h35m24.164989931s) [noise=true , address=145.79.227.24:4583 , piece_id=WYTXGXG77ZW4RGWX7YWDU2NH6R5CCPVBPQH3ISHUAJVKZJRONS5Q, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.298576738 (58h35m24.169659783s) [noise=true , address=145.79.227.28:4727 , piece_id=UYRK457JBF2AHCIXZ4XOIL7YHOVYZ7IPDUTB6WOZZUDEW5Q7NNCQ, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.301356571 (58h35m24.172439616s) [noise=true , address=145.79.227.25:4619 , piece_id=XBNGVRIQLQUWKKBF4ULL7ZJDDT75QLY7ES2J7WHYWRD2ZLHOGU2Q, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.326466765 (58h35m24.19754981s) [noise=true , address=145.79.227.16:4910 , piece_id=ETE7ZBHT5IJUOBWCA5BGLKC2ANDEUB2M3DOMXWWS2RBWSJQBARRQ, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.328633530 (58h35m24.199716586s) [noise=true , address=145.79.227.26:4653 , piece_id=DKW72PEWPIOHPYHLI54PBU7NR3HGUBLFM3MN4XQQFWIIZ4FHIUNA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.360508106 (58h35m24.231591151s) [noise=true , address=145.79.227.40:4051 , piece_id=7CQTGHFZ542WUHYX2CTOPN4PF2KDOOLDGV64THDBZCYOCXA3JWDA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.361116263 (58h35m24.232199308s) [noise=true , address=145.79.227.25:4617 , piece_id=S4ETCJ6WMCPXXQCYPPBBMHLMK2SVJV4S5VOKHGTDAGSGG6WRYH3Q, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.361695086 (58h35m24.232778131s) [noise=true , address=145.79.227.41:4087 , piece_id=XO6K3CXK7KZ5VEVXULHKQXLSSLIIWIX3VKQOSD7HY25WMNMCDLEQ, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.371520525 (58h35m24.24260356s) [noise=true , address=145.79.227.38:4977 , piece_id=73WXA33TWM4MFVTHDHW3PXJVABMCVMQYI3P42AHT3OYPKMXIVTUQ, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.381271893 (58h35m24.252354928s) [noise=true , address=145.79.227.23:4545 , piece_id=NBYCSU2KSHTAL223AHKHREQWKXUEQKJU46TY3SPYTKT5N5T2OUSA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.386403053 (58h35m24.257486098s) [noise=true , address=77.90.31.11:4554 , piece_id=BQP7XTRA2SBVINT4J7JAOF3ORNENBBIQN4ABMCEMQQ6FQL6QUOKA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.386427209 (58h35m24.257510254s) [noise=true , address=77.90.31.10:4518 , piece_id=6AR3NAFODP54SKG3SJSJSIQQFOYXO2C65WVNN4Z4XG426ZEOEDQA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.400489386 (58h35m24.271572431s) [noise=true , address=77.90.31.13:4628 , piece_id=LWTOCQRRA7HLI5ZPPRQ25IM22LMQNRR5H3TTOTSTGCFOYFZMABAA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.400513903 (58h35m24.271596938s) [seg_pos={0 0}, upload=51469]: Segment reached optimal threshold of 13 pieces. Dec 8 02:04:24.400813233 (58h35m24.271896288s) [noise=true , address=145.79.227.39:4013 , piece_id=UI22RDXJ2DY3RFNSJVNT43XQJ6B2S3EO6SFLYHXW4VPVQJSYYF6Q, seg_pos={0 0}, upload=51469]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b): protocol: write tcp 188.166.225.170:39434->145.79.227.39:4013: use of closed network connection; write tcp 188.166.225.170:39434->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39434->145.79.227.39:4013: use of closed network connection Dec 8 02:04:24.400825677 (58h35m24.271908722s) [noise=true , address=145.79.227.39:4013 , piece_id=UI22RDXJ2DY3RFNSJVNT43XQJ6B2S3EO6SFLYHXW4VPVQJSYYF6Q, seg_pos={0 0}, upload=51469]: Long tail cancellation due to optimal threshold Dec 8 02:04:24.400873027 (58h35m24.271956082s) [noise=true , address=145.79.227.22:4511 , piece_id=GGA463DAOSZCW7SQGZW2LGFRQVNUOUJ2L2GXRXLSDPC65KVPY3TA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG): protocol: write tcp 188.166.225.170:47926->145.79.227.22:4511: use of closed network connection; write tcp 188.166.225.170:47926->145.79.227.22:4511: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47926->145.79.227.22:4511: use of closed network connection Dec 8 02:04:24.400894248 (58h35m24.271977293s) [noise=true , address=145.79.227.22:4511 , piece_id=GGA463DAOSZCW7SQGZW2LGFRQVNUOUJ2L2GXRXLSDPC65KVPY3TA, seg_pos={0 0}, upload=51469]: Long tail cancellation due to optimal threshold Dec 8 02:04:24.401247129 (58h35m24.272330184s) [noise=true , address=145.79.227.23:4547 , piece_id=5Y6V5QWLCWNOLT2GDNUEIRSG75KGNGIFLBCBGSIT7XH46RZYVDGQ, seg_pos={0 0}, upload=51469]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW): protocol: read tcp 188.166.225.170:43344->145.79.227.23:4547: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 02:04:24.401260003 (58h35m24.272343059s) [noise=true , address=145.79.227.23:4547 , piece_id=5Y6V5QWLCWNOLT2GDNUEIRSG75KGNGIFLBCBGSIT7XH46RZYVDGQ, seg_pos={0 0}, upload=51469]: Long tail cancellation due to optimal threshold Dec 8 02:04:24.401285452 (58h35m24.272368497s) [seg_pos={0 0}, upload=51469]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 02:04:24.401382247 (58h35m24.272465292s) [upload=51469]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 02:04:24.401391394 (58h35m24.272474429s) [upload=51469]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 02:05:39.049922629 (58h36m38.921005704s) [upload=51469]: Done uploading segments... Dec 8 02:05:39.050003342 (58h36m38.921086377s) [upload=51469]: Upload finished. err: metaclient: context deadline exceeded Duration: 7.445421265s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 5.700626854s trace: https://jaeger.dev.storj.io/trace/4030003299248e86 FAIL (2025-12-08 02:23:53 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.700653294s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [.......................X......X................X....X.......] duration: 3.312323458s trace: https://jaeger.dev.storj.io/trace/491130a80ad889c4 FAIL (2025-12-08 02:23:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.312411597s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 3 out of 60 is failed (threshold=3) [....X................X.........X............................] duration: 6.328963444s trace: https://jaeger.dev.storj.io/trace/e7164007ac0fd55 FAIL (2025-12-08 02:23:45 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 6.328990736s core uplink us1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 4.950159513s trace: https://jaeger.dev.storj.io/trace/7af096edea7ffcf5 FAIL (2025-12-08 02:23:34 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.950192486s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [........................X......X................X....X......] duration: 3.434149667s trace: https://jaeger.dev.storj.io/trace/491130a80ad889c4 FAIL (2025-12-08 02:23:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.434178702s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 8.017678902s trace: https://jaeger.dev.storj.io/trace/27d92c196f473eec FAIL (2025-12-08 02:23:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 4891s) [upload=51469]: Got next segment. Inline: false Dec 8 02:04:21.143134547 (58h35m21.014217592s) [upload=51469]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 02:04:21.143141259 (58h35m21.014224304s) [upload=51469]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 02:04:22.267502706 (58h35m22.138585751s) [seg_pos={0 0}, upload=51469]: Begin upload segment... Dec 8 02:04:22.267624127 (58h35m22.138707172s) [seg_pos={0 0}, upload=51469]: Done begin upload segment. Dec 8 02:04:22.267698027 (58h35m22.138781073s) [noise=true , address=77.90.31.11:4554 , piece_id=BQP7XTRA2SBVINT4J7JAOF3ORNENBBIQN4ABMCEMQQ6FQL6QUOKA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.267712997 (58h35m22.138796042s) [noise=true , address=145.79.227.41:4087 , piece_id=XO6K3CXK7KZ5VEVXULHKQXLSSLIIWIX3VKQOSD7HY25WMNMCDLEQ, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.268434680 (58h35m22.139517715s) [noise=true , address=145.79.227.16:4910 , piece_id=ETE7ZBHT5IJUOBWCA5BGLKC2ANDEUB2M3DOMXWWS2RBWSJQBARRQ, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.268550331 (58h35m22.139633366s) [noise=true , address=145.79.227.26:4653 , piece_id=DKW72PEWPIOHPYHLI54PBU7NR3HGUBLFM3MN4XQQFWIIZ4FHIUNA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.268913111 (58h35m22.139996146s) [noise=true , address=145.79.227.23:4545 , piece_id=NBYCSU2KSHTAL223AHKHREQWKXUEQKJU46TY3SPYTKT5N5T2OUSA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.268924032 (58h35m22.140007067s) [noise=true , address=145.79.227.40:4051 , piece_id=7CQTGHFZ542WUHYX2CTOPN4PF2KDOOLDGV64THDBZCYOCXA3JWDA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.269398886 (58h35m22.140481921s) [noise=true , address=77.90.31.10:4518 , piece_id=6AR3NAFODP54SKG3SJSJSIQQFOYXO2C65WVNN4Z4XG426ZEOEDQA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.269414686 (58h35m22.140497721s) [noise=true , address=145.79.227.25:4619 , piece_id=XBNGVRIQLQUWKKBF4ULL7ZJDDT75QLY7ES2J7WHYWRD2ZLHOGU2Q, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.269753261 (58h35m22.140836296s) [noise=true , address=145.79.227.23:4547 , piece_id=5Y6V5QWLCWNOLT2GDNUEIRSG75KGNGIFLBCBGSIT7XH46RZYVDGQ, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.270016322 (58h35m22.141099367s) [noise=true , address=77.90.31.13:4628 , piece_id=LWTOCQRRA7HLI5ZPPRQ25IM22LMQNRR5H3TTOTSTGCFOYFZMABAA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.270259175 (58h35m22.14134222s) [noise=true , address=145.79.227.25:4617 , piece_id=S4ETCJ6WMCPXXQCYPPBBMHLMK2SVJV4S5VOKHGTDAGSGG6WRYH3Q, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.270655339 (58h35m22.141738374s) [noise=true , address=145.79.227.38:4977 , piece_id=73WXA33TWM4MFVTHDHW3PXJVABMCVMQYI3P42AHT3OYPKMXIVTUQ, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.271143158 (58h35m22.142246412s) [noise=true , address=145.79.227.28:4727 , piece_id=UYRK457JBF2AHCIXZ4XOIL7YHOVYZ7IPDUTB6WOZZUDEW5Q7NNCQ, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.271230565 (58h35m22.14231361s) [noise=true , address=145.79.227.39:4013 , piece_id=UI22RDXJ2DY3RFNSJVNT43XQJ6B2S3EO6SFLYHXW4VPVQJSYYF6Q, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.271662086 (58h35m22.142745121s) [noise=true , address=145.79.227.22:4511 , piece_id=GGA463DAOSZCW7SQGZW2LGFRQVNUOUJ2L2GXRXLSDPC65KVPY3TA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.272592658 (58h35m22.143675703s) [noise=true , address=145.79.227.24:4583 , piece_id=WYTXGXG77ZW4RGWX7YWDU2NH6R5CCPVBPQH3ISHUAJVKZJRONS5Q, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:24.009199123 (58h35m23.880282158s) [upload=51469]: Next returned nil segment Dec 8 02:04:24.009221235 (58h35m23.88030428s) [upload=51469]: Waiting for error group managing segments... Dec 8 02:04:24.293906896 (58h35m24.164989931s) [noise=true , address=145.79.227.24:4583 , piece_id=WYTXGXG77ZW4RGWX7YWDU2NH6R5CCPVBPQH3ISHUAJVKZJRONS5Q, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.298576738 (58h35m24.169659783s) [noise=true , address=145.79.227.28:4727 , piece_id=UYRK457JBF2AHCIXZ4XOIL7YHOVYZ7IPDUTB6WOZZUDEW5Q7NNCQ, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.301356571 (58h35m24.172439616s) [noise=true , address=145.79.227.25:4619 , piece_id=XBNGVRIQLQUWKKBF4ULL7ZJDDT75QLY7ES2J7WHYWRD2ZLHOGU2Q, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.326466765 (58h35m24.19754981s) [noise=true , address=145.79.227.16:4910 , piece_id=ETE7ZBHT5IJUOBWCA5BGLKC2ANDEUB2M3DOMXWWS2RBWSJQBARRQ, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.328633530 (58h35m24.199716586s) [noise=true , address=145.79.227.26:4653 , piece_id=DKW72PEWPIOHPYHLI54PBU7NR3HGUBLFM3MN4XQQFWIIZ4FHIUNA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.360508106 (58h35m24.231591151s) [noise=true , address=145.79.227.40:4051 , piece_id=7CQTGHFZ542WUHYX2CTOPN4PF2KDOOLDGV64THDBZCYOCXA3JWDA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.361116263 (58h35m24.232199308s) [noise=true , address=145.79.227.25:4617 , piece_id=S4ETCJ6WMCPXXQCYPPBBMHLMK2SVJV4S5VOKHGTDAGSGG6WRYH3Q, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.361695086 (58h35m24.232778131s) [noise=true , address=145.79.227.41:4087 , piece_id=XO6K3CXK7KZ5VEVXULHKQXLSSLIIWIX3VKQOSD7HY25WMNMCDLEQ, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.371520525 (58h35m24.24260356s) [noise=true , address=145.79.227.38:4977 , piece_id=73WXA33TWM4MFVTHDHW3PXJVABMCVMQYI3P42AHT3OYPKMXIVTUQ, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.381271893 (58h35m24.252354928s) [noise=true , address=145.79.227.23:4545 , piece_id=NBYCSU2KSHTAL223AHKHREQWKXUEQKJU46TY3SPYTKT5N5T2OUSA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.386403053 (58h35m24.257486098s) [noise=true , address=77.90.31.11:4554 , piece_id=BQP7XTRA2SBVINT4J7JAOF3ORNENBBIQN4ABMCEMQQ6FQL6QUOKA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.386427209 (58h35m24.257510254s) [noise=true , address=77.90.31.10:4518 , piece_id=6AR3NAFODP54SKG3SJSJSIQQFOYXO2C65WVNN4Z4XG426ZEOEDQA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.400489386 (58h35m24.271572431s) [noise=true , address=77.90.31.13:4628 , piece_id=LWTOCQRRA7HLI5ZPPRQ25IM22LMQNRR5H3TTOTSTGCFOYFZMABAA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.400513903 (58h35m24.271596938s) [seg_pos={0 0}, upload=51469]: Segment reached optimal threshold of 13 pieces. Dec 8 02:04:24.400813233 (58h35m24.271896288s) [noise=true , address=145.79.227.39:4013 , piece_id=UI22RDXJ2DY3RFNSJVNT43XQJ6B2S3EO6SFLYHXW4VPVQJSYYF6Q, seg_pos={0 0}, upload=51469]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b): protocol: write tcp 188.166.225.170:39434->145.79.227.39:4013: use of closed network connection; write tcp 188.166.225.170:39434->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39434->145.79.227.39:4013: use of closed network connection Dec 8 02:04:24.400825677 (58h35m24.271908722s) [noise=true , address=145.79.227.39:4013 , piece_id=UI22RDXJ2DY3RFNSJVNT43XQJ6B2S3EO6SFLYHXW4VPVQJSYYF6Q, seg_pos={0 0}, upload=51469]: Long tail cancellation due to optimal threshold Dec 8 02:04:24.400873027 (58h35m24.271956082s) [noise=true , address=145.79.227.22:4511 , piece_id=GGA463DAOSZCW7SQGZW2LGFRQVNUOUJ2L2GXRXLSDPC65KVPY3TA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG): protocol: write tcp 188.166.225.170:47926->145.79.227.22:4511: use of closed network connection; write tcp 188.166.225.170:47926->145.79.227.22:4511: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47926->145.79.227.22:4511: use of closed network connection Dec 8 02:04:24.400894248 (58h35m24.271977293s) [noise=true , address=145.79.227.22:4511 , piece_id=GGA463DAOSZCW7SQGZW2LGFRQVNUOUJ2L2GXRXLSDPC65KVPY3TA, seg_pos={0 0}, upload=51469]: Long tail cancellation due to optimal threshold Dec 8 02:04:24.401247129 (58h35m24.272330184s) [noise=true , address=145.79.227.23:4547 , piece_id=5Y6V5QWLCWNOLT2GDNUEIRSG75KGNGIFLBCBGSIT7XH46RZYVDGQ, seg_pos={0 0}, upload=51469]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW): protocol: read tcp 188.166.225.170:43344->145.79.227.23:4547: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 02:04:24.401260003 (58h35m24.272343059s) [noise=true , address=145.79.227.23:4547 , piece_id=5Y6V5QWLCWNOLT2GDNUEIRSG75KGNGIFLBCBGSIT7XH46RZYVDGQ, seg_pos={0 0}, upload=51469]: Long tail cancellation due to optimal threshold Dec 8 02:04:24.401285452 (58h35m24.272368497s) [seg_pos={0 0}, upload=51469]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 02:04:24.401382247 (58h35m24.272465292s) [upload=51469]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 02:04:24.401391394 (58h35m24.272474429s) [upload=51469]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 02:05:39.049922629 (58h36m38.921005704s) [upload=51469]: Done uploading segments... Dec 8 02:05:39.050003342 (58h36m38.921086377s) [upload=51469]: Upload finished. err: metaclient: context deadline exceeded Duration: 8.017708649s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 5.915973451s trace: https://jaeger.dev.storj.io/trace/57258282a8e974fd FAIL (2025-12-08 02:23:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== UA4TAJ2FSYO5S4NXECVMMMCXKFL5CAHPQBQJ6WREQ3YFC3Q, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.202906305 (58h26m7.07398934s) [noise=true , address=45.91.116.4:4945 , piece_id=6UNWQQAWWWWC2XVUCDRCTAYJDTHBXAZJMXA6KLCTOP5PYSAJCCGA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.203371201 (58h26m7.074454246s) [noise=true , address=145.79.227.11:4740 , piece_id=KL54LL2UQRT6MH2K3NQQJB3BKPX4QRWQMXRK27MW2U7CHFDCCWFA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.203861354 (58h26m7.07494439s) [noise=true , address=77.90.31.12:4621 , piece_id=S4FVJHEPIO2MVO56ZRIJ27RMTV723MERPLJLVD2T2GAO3T7K24TA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.204290401 (58h26m7.075373446s) [noise=true , address=145.79.227.34:4928 , piece_id=DRKF2DRSCBAYSZO66A2ZON2JS6HIJIT2YUWFZRS3W2CKFTV3NLVA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.204711363 (58h26m7.075794398s) [noise=true , address=145.79.227.15:4908 , piece_id=JIUZSMOAKBFCFAX6RQ52TCNZMYBREQCYOEM75U3H2QYR4WH3KWCA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.205118208 (58h26m7.076201253s) [noise=true , address=77.90.31.18:4825 , piece_id=EJ4LO4TCEFB4OQQLFOEUYGQKK462ZPXXEIIY5CHF2PERXWDGQNDA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.205554277 (58h26m7.076637323s) [noise=true , address=145.79.227.30:4786 , piece_id=VM65RUN3GUZZFUOHFYE7ZLPWU6COCQUMY3IK7GCNWPIUZAMEBR7Q, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.206116137 (58h26m7.077199182s) [noise=true , address=145.79.227.25:4627 , piece_id=K7UYU3W2CACUAQUUL72GG5F7DP5CHAHZ4M47TZG6NBN5Z6ZCDRBA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.206664562 (58h26m7.077747607s) [noise=true , address=77.90.31.16:4760 , piece_id=Q67LYTDJCAR2TFVSAIIYCUYM7D44JFPBLCSKRNLXH26EAW5UQ3GA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.207102556 (58h26m7.078185601s) [noise=true , address=45.91.116.7:4037 , piece_id=UBNFDSXQBISC5QQSNGAAZIR7LRE7FCYRVYN5MMOY3A2ALEOQBYQQ, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.207519911 (58h26m7.078602956s) [noise=true , address=77.90.31.11:4579 , piece_id=JINRNEVUAP2VI5Y4CR7ALWDQIAIAFX3SP2VVZSXDDG6XPUKASE3Q, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.207967393 (58h26m7.079050438s) [noise=true , address=77.90.31.10:4550 , piece_id=PDCZ7VXNHQJ6LYPG64GZRYP4MRBGVPXCWTXSRT3UV2N6HGMDE2KA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.208416427 (58h26m7.079499472s) [noise=true , address=45.91.116.6:4003 , piece_id=CRONTV2B2AVZTLKU3YNIEGQZSON7XHHL7S3K6KNNJV4YR5FHQ3PA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.208830746 (58h26m7.079913791s) [noise=true , address=45.91.116.12:4254 , piece_id=UCKU4TBXJC4PSYHRDPUXETXLAILDOWA5NN7RCVVNYLVUBY7FBIFA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.209255245 (58h26m7.080338291s) [noise=true , address=145.79.227.16:4933 , piece_id=LHF64L3SWXFVZ5X5D67I44WSJNKRFB5G4YP7DFHDWOAYE23II3OA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.209662921 (58h26m7.080745966s) [noise=true , address=145.79.227.18:4010 , piece_id=S2EOTEV4TBNNDOBN7TIX76AJE2WI5P5DM3JK2NJB25TISYLS7VSQ, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.214493680 (58h26m7.085576725s) [noise=true , address=77.90.31.10:4539 , piece_id=27YFK55QLZDEULYABQ3OBDSPDDEIUJ2MJSD7X3A4K6HRFJMRQJUA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:08.555121654 (58h26m8.426204699s) [upload=51329]: Next returned nil segment Dec 8 01:55:08.555145519 (58h26m8.426228564s) [upload=51329]: Waiting for error group managing segments... Dec 8 01:55:08.935193756 (58h26m8.806276791s) [noise=true , address=64.49.13.27:4957 , piece_id=7METNUA4TAJ2FSYO5S4NXECVMMMCXKFL5CAHPQBQJ6WREQ3YFC3Q, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.093051314 (58h26m8.964134359s) [noise=true , address=145.79.227.16:4933 , piece_id=LHF64L3SWXFVZ5X5D67I44WSJNKRFB5G4YP7DFHDWOAYE23II3OA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.098812273 (58h26m8.969895318s) [noise=true , address=145.79.227.18:4010 , piece_id=S2EOTEV4TBNNDOBN7TIX76AJE2WI5P5DM3JK2NJB25TISYLS7VSQ, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.102616718 (58h26m8.973699763s) [noise=true , address=145.79.227.25:4627 , piece_id=K7UYU3W2CACUAQUUL72GG5F7DP5CHAHZ4M47TZG6NBN5Z6ZCDRBA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.118902949 (58h26m8.989985994s) [noise=true , address=145.79.227.34:4928 , piece_id=DRKF2DRSCBAYSZO66A2ZON2JS6HIJIT2YUWFZRS3W2CKFTV3NLVA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.124783838 (58h26m8.995866883s) [noise=true , address=77.90.31.18:4825 , piece_id=EJ4LO4TCEFB4OQQLFOEUYGQKK462ZPXXEIIY5CHF2PERXWDGQNDA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.126718282 (58h26m8.997801327s) [noise=true , address=145.79.227.11:4740 , piece_id=KL54LL2UQRT6MH2K3NQQJB3BKPX4QRWQMXRK27MW2U7CHFDCCWFA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.145158485 (58h26m9.01624153s) [noise=true , address=77.90.31.10:4550 , piece_id=PDCZ7VXNHQJ6LYPG64GZRYP4MRBGVPXCWTXSRT3UV2N6HGMDE2KA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.156650047 (58h26m9.027733092s) [noise=true , address=145.79.227.15:4908 , piece_id=JIUZSMOAKBFCFAX6RQ52TCNZMYBREQCYOEM75U3H2QYR4WH3KWCA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.162020363 (58h26m9.033103398s) [noise=true , address=77.90.31.10:4539 , piece_id=27YFK55QLZDEULYABQ3OBDSPDDEIUJ2MJSD7X3A4K6HRFJMRQJUA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.198899907 (58h26m9.069982942s) [noise=true , address=77.90.31.12:4621 , piece_id=S4FVJHEPIO2MVO56ZRIJ27RMTV723MERPLJLVD2T2GAO3T7K24TA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.208468337 (58h26m9.079551382s) [noise=true , address=45.91.116.4:4945 , piece_id=6UNWQQAWWWWC2XVUCDRCTAYJDTHBXAZJMXA6KLCTOP5PYSAJCCGA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.212217697 (58h26m9.083300732s) [noise=true , address=45.91.116.12:4254 , piece_id=UCKU4TBXJC4PSYHRDPUXETXLAILDOWA5NN7RCVVNYLVUBY7FBIFA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.215832390 (58h26m9.086915435s) [noise=true , address=77.90.31.11:4579 , piece_id=JINRNEVUAP2VI5Y4CR7ALWDQIAIAFX3SP2VVZSXDDG6XPUKASE3Q, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.215862086 (58h26m9.086945161s) [seg_pos={0 0}, upload=51329]: Segment reached optimal threshold of 14 pieces. Dec 8 01:55:09.216078098 (58h26m9.087161133s) [noise=true , address=45.91.116.6:4003 , piece_id=CRONTV2B2AVZTLKU3YNIEGQZSON7XHHL7S3K6KNNJV4YR5FHQ3PA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1veqoBGw6a8AGMoigyuNJMjcEvCT93xRq2B1d2KNhs6aEmSqVF): protocol: expected piece hash; context canceled; EOF Dec 8 01:55:09.216085662 (58h26m9.087168707s) [noise=true , address=45.91.116.6:4003 , piece_id=CRONTV2B2AVZTLKU3YNIEGQZSON7XHHL7S3K6KNNJV4YR5FHQ3PA, seg_pos={0 0}, upload=51329]: Long tail cancellation due to optimal threshold Dec 8 01:55:09.216307705 (58h26m9.08739075s) [noise=true , address=145.79.227.30:4786 , piece_id=VM65RUN3GUZZFUOHFYE7ZLPWU6COCQUMY3IK7GCNWPIUZAMEBR7Q, seg_pos={0 0}, upload=51329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125TWU7R4pkmrUxsNLhgTD475aLbjUJ7psA6drJtE7oDqZXYJ4Q): protocol: expected piece hash; context canceled; EOF Dec 8 01:55:09.216313956 (58h26m9.087396991s) [noise=true , address=145.79.227.30:4786 , piece_id=VM65RUN3GUZZFUOHFYE7ZLPWU6COCQUMY3IK7GCNWPIUZAMEBR7Q, seg_pos={0 0}, upload=51329]: Long tail cancellation due to optimal threshold Dec 8 01:55:09.216414809 (58h26m9.087497844s) [noise=true , address=77.90.31.16:4760 , piece_id=Q67LYTDJCAR2TFVSAIIYCUYM7D44JFPBLCSKRNLXH26EAW5UQ3GA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1e6p1TD5CqUyWdZjkrxh2kRqMuyDiMUr9eimfeayDAKwhtBrZ8): protocol: expected piece hash; context canceled; EOF Dec 8 01:55:09.216420681 (58h26m9.087503726s) [noise=true , address=77.90.31.16:4760 , piece_id=Q67LYTDJCAR2TFVSAIIYCUYM7D44JFPBLCSKRNLXH26EAW5UQ3GA, seg_pos={0 0}, upload=51329]: Long tail cancellation due to optimal threshold Dec 8 01:55:09.216562861 (58h26m9.087645906s) [noise=true , address=45.91.116.7:4037 , piece_id=UBNFDSXQBISC5QQSNGAAZIR7LRE7FCYRVYN5MMOY3A2ALEOQBYQQ, seg_pos={0 0}, upload=51329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iNPwE2XSfXcaXAXcrH2jcyNGYeDiiWvzAcDQffrmjPwTVqVmK): protocol: expected piece hash; context canceled; EOF Dec 8 01:55:09.216570025 (58h26m9.08765307s) [noise=true , address=45.91.116.7:4037 , piece_id=UBNFDSXQBISC5QQSNGAAZIR7LRE7FCYRVYN5MMOY3A2ALEOQBYQQ, seg_pos={0 0}, upload=51329]: Long tail cancellation due to optimal threshold Dec 8 01:55:09.216585564 (58h26m9.087668609s) [seg_pos={0 0}, upload=51329]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 8 01:55:09.216654365 (58h26m9.0877374s) [upload=51329]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 01:55:09.216659064 (58h26m9.087742099s) [upload=51329]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 01:55:35.852213564 (58h26m35.723296629s) [upload=51329]: Done uploading segments... Dec 8 01:55:35.852273619 (58h26m35.723356664s) [upload=51329]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.916176377s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....X................X.........X...........................] duration: 5.863434235s trace: https://jaeger.dev.storj.io/trace/e7164007ac0fd55 FAIL (2025-12-08 02:23:00 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.863464393s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 5.788677195s trace: https://jaeger.dev.storj.io/trace/4030003299248e86 FAIL (2025-12-08 02:22:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.788696041s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 6.504217508s trace: https://jaeger.dev.storj.io/trace/27d92c196f473eec FAIL (2025-12-08 02:22:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 4891s) [upload=51469]: Got next segment. Inline: false Dec 8 02:04:21.143134547 (58h35m21.014217592s) [upload=51469]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 02:04:21.143141259 (58h35m21.014224304s) [upload=51469]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 02:04:22.267502706 (58h35m22.138585751s) [seg_pos={0 0}, upload=51469]: Begin upload segment... Dec 8 02:04:22.267624127 (58h35m22.138707172s) [seg_pos={0 0}, upload=51469]: Done begin upload segment. Dec 8 02:04:22.267698027 (58h35m22.138781073s) [noise=true , address=77.90.31.11:4554 , piece_id=BQP7XTRA2SBVINT4J7JAOF3ORNENBBIQN4ABMCEMQQ6FQL6QUOKA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.267712997 (58h35m22.138796042s) [noise=true , address=145.79.227.41:4087 , piece_id=XO6K3CXK7KZ5VEVXULHKQXLSSLIIWIX3VKQOSD7HY25WMNMCDLEQ, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.268434680 (58h35m22.139517715s) [noise=true , address=145.79.227.16:4910 , piece_id=ETE7ZBHT5IJUOBWCA5BGLKC2ANDEUB2M3DOMXWWS2RBWSJQBARRQ, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.268550331 (58h35m22.139633366s) [noise=true , address=145.79.227.26:4653 , piece_id=DKW72PEWPIOHPYHLI54PBU7NR3HGUBLFM3MN4XQQFWIIZ4FHIUNA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.268913111 (58h35m22.139996146s) [noise=true , address=145.79.227.23:4545 , piece_id=NBYCSU2KSHTAL223AHKHREQWKXUEQKJU46TY3SPYTKT5N5T2OUSA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.268924032 (58h35m22.140007067s) [noise=true , address=145.79.227.40:4051 , piece_id=7CQTGHFZ542WUHYX2CTOPN4PF2KDOOLDGV64THDBZCYOCXA3JWDA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.269398886 (58h35m22.140481921s) [noise=true , address=77.90.31.10:4518 , piece_id=6AR3NAFODP54SKG3SJSJSIQQFOYXO2C65WVNN4Z4XG426ZEOEDQA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.269414686 (58h35m22.140497721s) [noise=true , address=145.79.227.25:4619 , piece_id=XBNGVRIQLQUWKKBF4ULL7ZJDDT75QLY7ES2J7WHYWRD2ZLHOGU2Q, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.269753261 (58h35m22.140836296s) [noise=true , address=145.79.227.23:4547 , piece_id=5Y6V5QWLCWNOLT2GDNUEIRSG75KGNGIFLBCBGSIT7XH46RZYVDGQ, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.270016322 (58h35m22.141099367s) [noise=true , address=77.90.31.13:4628 , piece_id=LWTOCQRRA7HLI5ZPPRQ25IM22LMQNRR5H3TTOTSTGCFOYFZMABAA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.270259175 (58h35m22.14134222s) [noise=true , address=145.79.227.25:4617 , piece_id=S4ETCJ6WMCPXXQCYPPBBMHLMK2SVJV4S5VOKHGTDAGSGG6WRYH3Q, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.270655339 (58h35m22.141738374s) [noise=true , address=145.79.227.38:4977 , piece_id=73WXA33TWM4MFVTHDHW3PXJVABMCVMQYI3P42AHT3OYPKMXIVTUQ, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.271143158 (58h35m22.142246412s) [noise=true , address=145.79.227.28:4727 , piece_id=UYRK457JBF2AHCIXZ4XOIL7YHOVYZ7IPDUTB6WOZZUDEW5Q7NNCQ, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.271230565 (58h35m22.14231361s) [noise=true , address=145.79.227.39:4013 , piece_id=UI22RDXJ2DY3RFNSJVNT43XQJ6B2S3EO6SFLYHXW4VPVQJSYYF6Q, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.271662086 (58h35m22.142745121s) [noise=true , address=145.79.227.22:4511 , piece_id=GGA463DAOSZCW7SQGZW2LGFRQVNUOUJ2L2GXRXLSDPC65KVPY3TA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.272592658 (58h35m22.143675703s) [noise=true , address=145.79.227.24:4583 , piece_id=WYTXGXG77ZW4RGWX7YWDU2NH6R5CCPVBPQH3ISHUAJVKZJRONS5Q, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:24.009199123 (58h35m23.880282158s) [upload=51469]: Next returned nil segment Dec 8 02:04:24.009221235 (58h35m23.88030428s) [upload=51469]: Waiting for error group managing segments... Dec 8 02:04:24.293906896 (58h35m24.164989931s) [noise=true , address=145.79.227.24:4583 , piece_id=WYTXGXG77ZW4RGWX7YWDU2NH6R5CCPVBPQH3ISHUAJVKZJRONS5Q, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.298576738 (58h35m24.169659783s) [noise=true , address=145.79.227.28:4727 , piece_id=UYRK457JBF2AHCIXZ4XOIL7YHOVYZ7IPDUTB6WOZZUDEW5Q7NNCQ, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.301356571 (58h35m24.172439616s) [noise=true , address=145.79.227.25:4619 , piece_id=XBNGVRIQLQUWKKBF4ULL7ZJDDT75QLY7ES2J7WHYWRD2ZLHOGU2Q, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.326466765 (58h35m24.19754981s) [noise=true , address=145.79.227.16:4910 , piece_id=ETE7ZBHT5IJUOBWCA5BGLKC2ANDEUB2M3DOMXWWS2RBWSJQBARRQ, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.328633530 (58h35m24.199716586s) [noise=true , address=145.79.227.26:4653 , piece_id=DKW72PEWPIOHPYHLI54PBU7NR3HGUBLFM3MN4XQQFWIIZ4FHIUNA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.360508106 (58h35m24.231591151s) [noise=true , address=145.79.227.40:4051 , piece_id=7CQTGHFZ542WUHYX2CTOPN4PF2KDOOLDGV64THDBZCYOCXA3JWDA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.361116263 (58h35m24.232199308s) [noise=true , address=145.79.227.25:4617 , piece_id=S4ETCJ6WMCPXXQCYPPBBMHLMK2SVJV4S5VOKHGTDAGSGG6WRYH3Q, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.361695086 (58h35m24.232778131s) [noise=true , address=145.79.227.41:4087 , piece_id=XO6K3CXK7KZ5VEVXULHKQXLSSLIIWIX3VKQOSD7HY25WMNMCDLEQ, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.371520525 (58h35m24.24260356s) [noise=true , address=145.79.227.38:4977 , piece_id=73WXA33TWM4MFVTHDHW3PXJVABMCVMQYI3P42AHT3OYPKMXIVTUQ, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.381271893 (58h35m24.252354928s) [noise=true , address=145.79.227.23:4545 , piece_id=NBYCSU2KSHTAL223AHKHREQWKXUEQKJU46TY3SPYTKT5N5T2OUSA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.386403053 (58h35m24.257486098s) [noise=true , address=77.90.31.11:4554 , piece_id=BQP7XTRA2SBVINT4J7JAOF3ORNENBBIQN4ABMCEMQQ6FQL6QUOKA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.386427209 (58h35m24.257510254s) [noise=true , address=77.90.31.10:4518 , piece_id=6AR3NAFODP54SKG3SJSJSIQQFOYXO2C65WVNN4Z4XG426ZEOEDQA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.400489386 (58h35m24.271572431s) [noise=true , address=77.90.31.13:4628 , piece_id=LWTOCQRRA7HLI5ZPPRQ25IM22LMQNRR5H3TTOTSTGCFOYFZMABAA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.400513903 (58h35m24.271596938s) [seg_pos={0 0}, upload=51469]: Segment reached optimal threshold of 13 pieces. Dec 8 02:04:24.400813233 (58h35m24.271896288s) [noise=true , address=145.79.227.39:4013 , piece_id=UI22RDXJ2DY3RFNSJVNT43XQJ6B2S3EO6SFLYHXW4VPVQJSYYF6Q, seg_pos={0 0}, upload=51469]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b): protocol: write tcp 188.166.225.170:39434->145.79.227.39:4013: use of closed network connection; write tcp 188.166.225.170:39434->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39434->145.79.227.39:4013: use of closed network connection Dec 8 02:04:24.400825677 (58h35m24.271908722s) [noise=true , address=145.79.227.39:4013 , piece_id=UI22RDXJ2DY3RFNSJVNT43XQJ6B2S3EO6SFLYHXW4VPVQJSYYF6Q, seg_pos={0 0}, upload=51469]: Long tail cancellation due to optimal threshold Dec 8 02:04:24.400873027 (58h35m24.271956082s) [noise=true , address=145.79.227.22:4511 , piece_id=GGA463DAOSZCW7SQGZW2LGFRQVNUOUJ2L2GXRXLSDPC65KVPY3TA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG): protocol: write tcp 188.166.225.170:47926->145.79.227.22:4511: use of closed network connection; write tcp 188.166.225.170:47926->145.79.227.22:4511: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47926->145.79.227.22:4511: use of closed network connection Dec 8 02:04:24.400894248 (58h35m24.271977293s) [noise=true , address=145.79.227.22:4511 , piece_id=GGA463DAOSZCW7SQGZW2LGFRQVNUOUJ2L2GXRXLSDPC65KVPY3TA, seg_pos={0 0}, upload=51469]: Long tail cancellation due to optimal threshold Dec 8 02:04:24.401247129 (58h35m24.272330184s) [noise=true , address=145.79.227.23:4547 , piece_id=5Y6V5QWLCWNOLT2GDNUEIRSG75KGNGIFLBCBGSIT7XH46RZYVDGQ, seg_pos={0 0}, upload=51469]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW): protocol: read tcp 188.166.225.170:43344->145.79.227.23:4547: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 02:04:24.401260003 (58h35m24.272343059s) [noise=true , address=145.79.227.23:4547 , piece_id=5Y6V5QWLCWNOLT2GDNUEIRSG75KGNGIFLBCBGSIT7XH46RZYVDGQ, seg_pos={0 0}, upload=51469]: Long tail cancellation due to optimal threshold Dec 8 02:04:24.401285452 (58h35m24.272368497s) [seg_pos={0 0}, upload=51469]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 02:04:24.401382247 (58h35m24.272465292s) [upload=51469]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 02:04:24.401391394 (58h35m24.272474429s) [upload=51469]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 02:05:39.049922629 (58h36m38.921005704s) [upload=51469]: Done uploading segments... Dec 8 02:05:39.050003342 (58h36m38.921086377s) [upload=51469]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.5042499s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 1m0.002595773s trace: https://jaeger.dev.storj.io/trace/4030003299248e86 FAIL (2025-12-08 02:22:45 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.002613056s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [.........................X......X................X....X.....] duration: 3.442667265s trace: https://jaeger.dev.storj.io/trace/491130a80ad889c4 FAIL (2025-12-08 02:22:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.442692393s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 8.501142855s trace: https://jaeger.dev.storj.io/trace/27d92c196f473eec FAIL (2025-12-08 02:22:29 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 4891s) [upload=51469]: Got next segment. Inline: false Dec 8 02:04:21.143134547 (58h35m21.014217592s) [upload=51469]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 02:04:21.143141259 (58h35m21.014224304s) [upload=51469]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 02:04:22.267502706 (58h35m22.138585751s) [seg_pos={0 0}, upload=51469]: Begin upload segment... Dec 8 02:04:22.267624127 (58h35m22.138707172s) [seg_pos={0 0}, upload=51469]: Done begin upload segment. Dec 8 02:04:22.267698027 (58h35m22.138781073s) [noise=true , address=77.90.31.11:4554 , piece_id=BQP7XTRA2SBVINT4J7JAOF3ORNENBBIQN4ABMCEMQQ6FQL6QUOKA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.267712997 (58h35m22.138796042s) [noise=true , address=145.79.227.41:4087 , piece_id=XO6K3CXK7KZ5VEVXULHKQXLSSLIIWIX3VKQOSD7HY25WMNMCDLEQ, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.268434680 (58h35m22.139517715s) [noise=true , address=145.79.227.16:4910 , piece_id=ETE7ZBHT5IJUOBWCA5BGLKC2ANDEUB2M3DOMXWWS2RBWSJQBARRQ, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.268550331 (58h35m22.139633366s) [noise=true , address=145.79.227.26:4653 , piece_id=DKW72PEWPIOHPYHLI54PBU7NR3HGUBLFM3MN4XQQFWIIZ4FHIUNA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.268913111 (58h35m22.139996146s) [noise=true , address=145.79.227.23:4545 , piece_id=NBYCSU2KSHTAL223AHKHREQWKXUEQKJU46TY3SPYTKT5N5T2OUSA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.268924032 (58h35m22.140007067s) [noise=true , address=145.79.227.40:4051 , piece_id=7CQTGHFZ542WUHYX2CTOPN4PF2KDOOLDGV64THDBZCYOCXA3JWDA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.269398886 (58h35m22.140481921s) [noise=true , address=77.90.31.10:4518 , piece_id=6AR3NAFODP54SKG3SJSJSIQQFOYXO2C65WVNN4Z4XG426ZEOEDQA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.269414686 (58h35m22.140497721s) [noise=true , address=145.79.227.25:4619 , piece_id=XBNGVRIQLQUWKKBF4ULL7ZJDDT75QLY7ES2J7WHYWRD2ZLHOGU2Q, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.269753261 (58h35m22.140836296s) [noise=true , address=145.79.227.23:4547 , piece_id=5Y6V5QWLCWNOLT2GDNUEIRSG75KGNGIFLBCBGSIT7XH46RZYVDGQ, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.270016322 (58h35m22.141099367s) [noise=true , address=77.90.31.13:4628 , piece_id=LWTOCQRRA7HLI5ZPPRQ25IM22LMQNRR5H3TTOTSTGCFOYFZMABAA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.270259175 (58h35m22.14134222s) [noise=true , address=145.79.227.25:4617 , piece_id=S4ETCJ6WMCPXXQCYPPBBMHLMK2SVJV4S5VOKHGTDAGSGG6WRYH3Q, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.270655339 (58h35m22.141738374s) [noise=true , address=145.79.227.38:4977 , piece_id=73WXA33TWM4MFVTHDHW3PXJVABMCVMQYI3P42AHT3OYPKMXIVTUQ, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.271143158 (58h35m22.142246412s) [noise=true , address=145.79.227.28:4727 , piece_id=UYRK457JBF2AHCIXZ4XOIL7YHOVYZ7IPDUTB6WOZZUDEW5Q7NNCQ, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.271230565 (58h35m22.14231361s) [noise=true , address=145.79.227.39:4013 , piece_id=UI22RDXJ2DY3RFNSJVNT43XQJ6B2S3EO6SFLYHXW4VPVQJSYYF6Q, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.271662086 (58h35m22.142745121s) [noise=true , address=145.79.227.22:4511 , piece_id=GGA463DAOSZCW7SQGZW2LGFRQVNUOUJ2L2GXRXLSDPC65KVPY3TA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.272592658 (58h35m22.143675703s) [noise=true , address=145.79.227.24:4583 , piece_id=WYTXGXG77ZW4RGWX7YWDU2NH6R5CCPVBPQH3ISHUAJVKZJRONS5Q, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:24.009199123 (58h35m23.880282158s) [upload=51469]: Next returned nil segment Dec 8 02:04:24.009221235 (58h35m23.88030428s) [upload=51469]: Waiting for error group managing segments... Dec 8 02:04:24.293906896 (58h35m24.164989931s) [noise=true , address=145.79.227.24:4583 , piece_id=WYTXGXG77ZW4RGWX7YWDU2NH6R5CCPVBPQH3ISHUAJVKZJRONS5Q, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.298576738 (58h35m24.169659783s) [noise=true , address=145.79.227.28:4727 , piece_id=UYRK457JBF2AHCIXZ4XOIL7YHOVYZ7IPDUTB6WOZZUDEW5Q7NNCQ, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.301356571 (58h35m24.172439616s) [noise=true , address=145.79.227.25:4619 , piece_id=XBNGVRIQLQUWKKBF4ULL7ZJDDT75QLY7ES2J7WHYWRD2ZLHOGU2Q, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.326466765 (58h35m24.19754981s) [noise=true , address=145.79.227.16:4910 , piece_id=ETE7ZBHT5IJUOBWCA5BGLKC2ANDEUB2M3DOMXWWS2RBWSJQBARRQ, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.328633530 (58h35m24.199716586s) [noise=true , address=145.79.227.26:4653 , piece_id=DKW72PEWPIOHPYHLI54PBU7NR3HGUBLFM3MN4XQQFWIIZ4FHIUNA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.360508106 (58h35m24.231591151s) [noise=true , address=145.79.227.40:4051 , piece_id=7CQTGHFZ542WUHYX2CTOPN4PF2KDOOLDGV64THDBZCYOCXA3JWDA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.361116263 (58h35m24.232199308s) [noise=true , address=145.79.227.25:4617 , piece_id=S4ETCJ6WMCPXXQCYPPBBMHLMK2SVJV4S5VOKHGTDAGSGG6WRYH3Q, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.361695086 (58h35m24.232778131s) [noise=true , address=145.79.227.41:4087 , piece_id=XO6K3CXK7KZ5VEVXULHKQXLSSLIIWIX3VKQOSD7HY25WMNMCDLEQ, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.371520525 (58h35m24.24260356s) [noise=true , address=145.79.227.38:4977 , piece_id=73WXA33TWM4MFVTHDHW3PXJVABMCVMQYI3P42AHT3OYPKMXIVTUQ, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.381271893 (58h35m24.252354928s) [noise=true , address=145.79.227.23:4545 , piece_id=NBYCSU2KSHTAL223AHKHREQWKXUEQKJU46TY3SPYTKT5N5T2OUSA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.386403053 (58h35m24.257486098s) [noise=true , address=77.90.31.11:4554 , piece_id=BQP7XTRA2SBVINT4J7JAOF3ORNENBBIQN4ABMCEMQQ6FQL6QUOKA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.386427209 (58h35m24.257510254s) [noise=true , address=77.90.31.10:4518 , piece_id=6AR3NAFODP54SKG3SJSJSIQQFOYXO2C65WVNN4Z4XG426ZEOEDQA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.400489386 (58h35m24.271572431s) [noise=true , address=77.90.31.13:4628 , piece_id=LWTOCQRRA7HLI5ZPPRQ25IM22LMQNRR5H3TTOTSTGCFOYFZMABAA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.400513903 (58h35m24.271596938s) [seg_pos={0 0}, upload=51469]: Segment reached optimal threshold of 13 pieces. Dec 8 02:04:24.400813233 (58h35m24.271896288s) [noise=true , address=145.79.227.39:4013 , piece_id=UI22RDXJ2DY3RFNSJVNT43XQJ6B2S3EO6SFLYHXW4VPVQJSYYF6Q, seg_pos={0 0}, upload=51469]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b): protocol: write tcp 188.166.225.170:39434->145.79.227.39:4013: use of closed network connection; write tcp 188.166.225.170:39434->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39434->145.79.227.39:4013: use of closed network connection Dec 8 02:04:24.400825677 (58h35m24.271908722s) [noise=true , address=145.79.227.39:4013 , piece_id=UI22RDXJ2DY3RFNSJVNT43XQJ6B2S3EO6SFLYHXW4VPVQJSYYF6Q, seg_pos={0 0}, upload=51469]: Long tail cancellation due to optimal threshold Dec 8 02:04:24.400873027 (58h35m24.271956082s) [noise=true , address=145.79.227.22:4511 , piece_id=GGA463DAOSZCW7SQGZW2LGFRQVNUOUJ2L2GXRXLSDPC65KVPY3TA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG): protocol: write tcp 188.166.225.170:47926->145.79.227.22:4511: use of closed network connection; write tcp 188.166.225.170:47926->145.79.227.22:4511: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47926->145.79.227.22:4511: use of closed network connection Dec 8 02:04:24.400894248 (58h35m24.271977293s) [noise=true , address=145.79.227.22:4511 , piece_id=GGA463DAOSZCW7SQGZW2LGFRQVNUOUJ2L2GXRXLSDPC65KVPY3TA, seg_pos={0 0}, upload=51469]: Long tail cancellation due to optimal threshold Dec 8 02:04:24.401247129 (58h35m24.272330184s) [noise=true , address=145.79.227.23:4547 , piece_id=5Y6V5QWLCWNOLT2GDNUEIRSG75KGNGIFLBCBGSIT7XH46RZYVDGQ, seg_pos={0 0}, upload=51469]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW): protocol: read tcp 188.166.225.170:43344->145.79.227.23:4547: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 02:04:24.401260003 (58h35m24.272343059s) [noise=true , address=145.79.227.23:4547 , piece_id=5Y6V5QWLCWNOLT2GDNUEIRSG75KGNGIFLBCBGSIT7XH46RZYVDGQ, seg_pos={0 0}, upload=51469]: Long tail cancellation due to optimal threshold Dec 8 02:04:24.401285452 (58h35m24.272368497s) [seg_pos={0 0}, upload=51469]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 02:04:24.401382247 (58h35m24.272465292s) [upload=51469]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 02:04:24.401391394 (58h35m24.272474429s) [upload=51469]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 02:05:39.049922629 (58h36m38.921005704s) [upload=51469]: Done uploading segments... Dec 8 02:05:39.050003342 (58h36m38.921086377s) [upload=51469]: Upload finished. err: metaclient: context deadline exceeded Duration: 8.501174085s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 5.218811755s trace: https://jaeger.dev.storj.io/trace/7af096edea7ffcf5 FAIL (2025-12-08 02:22:22 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.218841171s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 6.675325959s trace: https://jaeger.dev.storj.io/trace/57258282a8e974fd FAIL (2025-12-08 02:22:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== UA4TAJ2FSYO5S4NXECVMMMCXKFL5CAHPQBQJ6WREQ3YFC3Q, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.202906305 (58h26m7.07398934s) [noise=true , address=45.91.116.4:4945 , piece_id=6UNWQQAWWWWC2XVUCDRCTAYJDTHBXAZJMXA6KLCTOP5PYSAJCCGA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.203371201 (58h26m7.074454246s) [noise=true , address=145.79.227.11:4740 , piece_id=KL54LL2UQRT6MH2K3NQQJB3BKPX4QRWQMXRK27MW2U7CHFDCCWFA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.203861354 (58h26m7.07494439s) [noise=true , address=77.90.31.12:4621 , piece_id=S4FVJHEPIO2MVO56ZRIJ27RMTV723MERPLJLVD2T2GAO3T7K24TA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.204290401 (58h26m7.075373446s) [noise=true , address=145.79.227.34:4928 , piece_id=DRKF2DRSCBAYSZO66A2ZON2JS6HIJIT2YUWFZRS3W2CKFTV3NLVA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.204711363 (58h26m7.075794398s) [noise=true , address=145.79.227.15:4908 , piece_id=JIUZSMOAKBFCFAX6RQ52TCNZMYBREQCYOEM75U3H2QYR4WH3KWCA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.205118208 (58h26m7.076201253s) [noise=true , address=77.90.31.18:4825 , piece_id=EJ4LO4TCEFB4OQQLFOEUYGQKK462ZPXXEIIY5CHF2PERXWDGQNDA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.205554277 (58h26m7.076637323s) [noise=true , address=145.79.227.30:4786 , piece_id=VM65RUN3GUZZFUOHFYE7ZLPWU6COCQUMY3IK7GCNWPIUZAMEBR7Q, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.206116137 (58h26m7.077199182s) [noise=true , address=145.79.227.25:4627 , piece_id=K7UYU3W2CACUAQUUL72GG5F7DP5CHAHZ4M47TZG6NBN5Z6ZCDRBA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.206664562 (58h26m7.077747607s) [noise=true , address=77.90.31.16:4760 , piece_id=Q67LYTDJCAR2TFVSAIIYCUYM7D44JFPBLCSKRNLXH26EAW5UQ3GA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.207102556 (58h26m7.078185601s) [noise=true , address=45.91.116.7:4037 , piece_id=UBNFDSXQBISC5QQSNGAAZIR7LRE7FCYRVYN5MMOY3A2ALEOQBYQQ, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.207519911 (58h26m7.078602956s) [noise=true , address=77.90.31.11:4579 , piece_id=JINRNEVUAP2VI5Y4CR7ALWDQIAIAFX3SP2VVZSXDDG6XPUKASE3Q, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.207967393 (58h26m7.079050438s) [noise=true , address=77.90.31.10:4550 , piece_id=PDCZ7VXNHQJ6LYPG64GZRYP4MRBGVPXCWTXSRT3UV2N6HGMDE2KA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.208416427 (58h26m7.079499472s) [noise=true , address=45.91.116.6:4003 , piece_id=CRONTV2B2AVZTLKU3YNIEGQZSON7XHHL7S3K6KNNJV4YR5FHQ3PA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.208830746 (58h26m7.079913791s) [noise=true , address=45.91.116.12:4254 , piece_id=UCKU4TBXJC4PSYHRDPUXETXLAILDOWA5NN7RCVVNYLVUBY7FBIFA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.209255245 (58h26m7.080338291s) [noise=true , address=145.79.227.16:4933 , piece_id=LHF64L3SWXFVZ5X5D67I44WSJNKRFB5G4YP7DFHDWOAYE23II3OA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.209662921 (58h26m7.080745966s) [noise=true , address=145.79.227.18:4010 , piece_id=S2EOTEV4TBNNDOBN7TIX76AJE2WI5P5DM3JK2NJB25TISYLS7VSQ, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.214493680 (58h26m7.085576725s) [noise=true , address=77.90.31.10:4539 , piece_id=27YFK55QLZDEULYABQ3OBDSPDDEIUJ2MJSD7X3A4K6HRFJMRQJUA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:08.555121654 (58h26m8.426204699s) [upload=51329]: Next returned nil segment Dec 8 01:55:08.555145519 (58h26m8.426228564s) [upload=51329]: Waiting for error group managing segments... Dec 8 01:55:08.935193756 (58h26m8.806276791s) [noise=true , address=64.49.13.27:4957 , piece_id=7METNUA4TAJ2FSYO5S4NXECVMMMCXKFL5CAHPQBQJ6WREQ3YFC3Q, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.093051314 (58h26m8.964134359s) [noise=true , address=145.79.227.16:4933 , piece_id=LHF64L3SWXFVZ5X5D67I44WSJNKRFB5G4YP7DFHDWOAYE23II3OA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.098812273 (58h26m8.969895318s) [noise=true , address=145.79.227.18:4010 , piece_id=S2EOTEV4TBNNDOBN7TIX76AJE2WI5P5DM3JK2NJB25TISYLS7VSQ, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.102616718 (58h26m8.973699763s) [noise=true , address=145.79.227.25:4627 , piece_id=K7UYU3W2CACUAQUUL72GG5F7DP5CHAHZ4M47TZG6NBN5Z6ZCDRBA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.118902949 (58h26m8.989985994s) [noise=true , address=145.79.227.34:4928 , piece_id=DRKF2DRSCBAYSZO66A2ZON2JS6HIJIT2YUWFZRS3W2CKFTV3NLVA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.124783838 (58h26m8.995866883s) [noise=true , address=77.90.31.18:4825 , piece_id=EJ4LO4TCEFB4OQQLFOEUYGQKK462ZPXXEIIY5CHF2PERXWDGQNDA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.126718282 (58h26m8.997801327s) [noise=true , address=145.79.227.11:4740 , piece_id=KL54LL2UQRT6MH2K3NQQJB3BKPX4QRWQMXRK27MW2U7CHFDCCWFA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.145158485 (58h26m9.01624153s) [noise=true , address=77.90.31.10:4550 , piece_id=PDCZ7VXNHQJ6LYPG64GZRYP4MRBGVPXCWTXSRT3UV2N6HGMDE2KA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.156650047 (58h26m9.027733092s) [noise=true , address=145.79.227.15:4908 , piece_id=JIUZSMOAKBFCFAX6RQ52TCNZMYBREQCYOEM75U3H2QYR4WH3KWCA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.162020363 (58h26m9.033103398s) [noise=true , address=77.90.31.10:4539 , piece_id=27YFK55QLZDEULYABQ3OBDSPDDEIUJ2MJSD7X3A4K6HRFJMRQJUA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.198899907 (58h26m9.069982942s) [noise=true , address=77.90.31.12:4621 , piece_id=S4FVJHEPIO2MVO56ZRIJ27RMTV723MERPLJLVD2T2GAO3T7K24TA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.208468337 (58h26m9.079551382s) [noise=true , address=45.91.116.4:4945 , piece_id=6UNWQQAWWWWC2XVUCDRCTAYJDTHBXAZJMXA6KLCTOP5PYSAJCCGA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.212217697 (58h26m9.083300732s) [noise=true , address=45.91.116.12:4254 , piece_id=UCKU4TBXJC4PSYHRDPUXETXLAILDOWA5NN7RCVVNYLVUBY7FBIFA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.215832390 (58h26m9.086915435s) [noise=true , address=77.90.31.11:4579 , piece_id=JINRNEVUAP2VI5Y4CR7ALWDQIAIAFX3SP2VVZSXDDG6XPUKASE3Q, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.215862086 (58h26m9.086945161s) [seg_pos={0 0}, upload=51329]: Segment reached optimal threshold of 14 pieces. Dec 8 01:55:09.216078098 (58h26m9.087161133s) [noise=true , address=45.91.116.6:4003 , piece_id=CRONTV2B2AVZTLKU3YNIEGQZSON7XHHL7S3K6KNNJV4YR5FHQ3PA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1veqoBGw6a8AGMoigyuNJMjcEvCT93xRq2B1d2KNhs6aEmSqVF): protocol: expected piece hash; context canceled; EOF Dec 8 01:55:09.216085662 (58h26m9.087168707s) [noise=true , address=45.91.116.6:4003 , piece_id=CRONTV2B2AVZTLKU3YNIEGQZSON7XHHL7S3K6KNNJV4YR5FHQ3PA, seg_pos={0 0}, upload=51329]: Long tail cancellation due to optimal threshold Dec 8 01:55:09.216307705 (58h26m9.08739075s) [noise=true , address=145.79.227.30:4786 , piece_id=VM65RUN3GUZZFUOHFYE7ZLPWU6COCQUMY3IK7GCNWPIUZAMEBR7Q, seg_pos={0 0}, upload=51329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125TWU7R4pkmrUxsNLhgTD475aLbjUJ7psA6drJtE7oDqZXYJ4Q): protocol: expected piece hash; context canceled; EOF Dec 8 01:55:09.216313956 (58h26m9.087396991s) [noise=true , address=145.79.227.30:4786 , piece_id=VM65RUN3GUZZFUOHFYE7ZLPWU6COCQUMY3IK7GCNWPIUZAMEBR7Q, seg_pos={0 0}, upload=51329]: Long tail cancellation due to optimal threshold Dec 8 01:55:09.216414809 (58h26m9.087497844s) [noise=true , address=77.90.31.16:4760 , piece_id=Q67LYTDJCAR2TFVSAIIYCUYM7D44JFPBLCSKRNLXH26EAW5UQ3GA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1e6p1TD5CqUyWdZjkrxh2kRqMuyDiMUr9eimfeayDAKwhtBrZ8): protocol: expected piece hash; context canceled; EOF Dec 8 01:55:09.216420681 (58h26m9.087503726s) [noise=true , address=77.90.31.16:4760 , piece_id=Q67LYTDJCAR2TFVSAIIYCUYM7D44JFPBLCSKRNLXH26EAW5UQ3GA, seg_pos={0 0}, upload=51329]: Long tail cancellation due to optimal threshold Dec 8 01:55:09.216562861 (58h26m9.087645906s) [noise=true , address=45.91.116.7:4037 , piece_id=UBNFDSXQBISC5QQSNGAAZIR7LRE7FCYRVYN5MMOY3A2ALEOQBYQQ, seg_pos={0 0}, upload=51329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iNPwE2XSfXcaXAXcrH2jcyNGYeDiiWvzAcDQffrmjPwTVqVmK): protocol: expected piece hash; context canceled; EOF Dec 8 01:55:09.216570025 (58h26m9.08765307s) [noise=true , address=45.91.116.7:4037 , piece_id=UBNFDSXQBISC5QQSNGAAZIR7LRE7FCYRVYN5MMOY3A2ALEOQBYQQ, seg_pos={0 0}, upload=51329]: Long tail cancellation due to optimal threshold Dec 8 01:55:09.216585564 (58h26m9.087668609s) [seg_pos={0 0}, upload=51329]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 8 01:55:09.216654365 (58h26m9.0877374s) [upload=51329]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 01:55:09.216659064 (58h26m9.087742099s) [upload=51329]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 01:55:35.852213564 (58h26m35.723296629s) [upload=51329]: Done uploading segments... Dec 8 01:55:35.852273619 (58h26m35.723356664s) [upload=51329]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.675342951s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 3 out of 60 is failed (threshold=3) [......X................X.........X..........................] duration: 6.418269357s trace: https://jaeger.dev.storj.io/trace/e7164007ac0fd55 FAIL (2025-12-08 02:22:03 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 6.418303783s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......X................X.........X.........................] duration: 5.919257476s trace: https://jaeger.dev.storj.io/trace/e7164007ac0fd55 FAIL (2025-12-08 02:21:45 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.919285389s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 5.927288537s trace: https://jaeger.dev.storj.io/trace/57258282a8e974fd FAIL (2025-12-08 02:21:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== UA4TAJ2FSYO5S4NXECVMMMCXKFL5CAHPQBQJ6WREQ3YFC3Q, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.202906305 (58h26m7.07398934s) [noise=true , address=45.91.116.4:4945 , piece_id=6UNWQQAWWWWC2XVUCDRCTAYJDTHBXAZJMXA6KLCTOP5PYSAJCCGA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.203371201 (58h26m7.074454246s) [noise=true , address=145.79.227.11:4740 , piece_id=KL54LL2UQRT6MH2K3NQQJB3BKPX4QRWQMXRK27MW2U7CHFDCCWFA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.203861354 (58h26m7.07494439s) [noise=true , address=77.90.31.12:4621 , piece_id=S4FVJHEPIO2MVO56ZRIJ27RMTV723MERPLJLVD2T2GAO3T7K24TA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.204290401 (58h26m7.075373446s) [noise=true , address=145.79.227.34:4928 , piece_id=DRKF2DRSCBAYSZO66A2ZON2JS6HIJIT2YUWFZRS3W2CKFTV3NLVA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.204711363 (58h26m7.075794398s) [noise=true , address=145.79.227.15:4908 , piece_id=JIUZSMOAKBFCFAX6RQ52TCNZMYBREQCYOEM75U3H2QYR4WH3KWCA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.205118208 (58h26m7.076201253s) [noise=true , address=77.90.31.18:4825 , piece_id=EJ4LO4TCEFB4OQQLFOEUYGQKK462ZPXXEIIY5CHF2PERXWDGQNDA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.205554277 (58h26m7.076637323s) [noise=true , address=145.79.227.30:4786 , piece_id=VM65RUN3GUZZFUOHFYE7ZLPWU6COCQUMY3IK7GCNWPIUZAMEBR7Q, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.206116137 (58h26m7.077199182s) [noise=true , address=145.79.227.25:4627 , piece_id=K7UYU3W2CACUAQUUL72GG5F7DP5CHAHZ4M47TZG6NBN5Z6ZCDRBA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.206664562 (58h26m7.077747607s) [noise=true , address=77.90.31.16:4760 , piece_id=Q67LYTDJCAR2TFVSAIIYCUYM7D44JFPBLCSKRNLXH26EAW5UQ3GA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.207102556 (58h26m7.078185601s) [noise=true , address=45.91.116.7:4037 , piece_id=UBNFDSXQBISC5QQSNGAAZIR7LRE7FCYRVYN5MMOY3A2ALEOQBYQQ, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.207519911 (58h26m7.078602956s) [noise=true , address=77.90.31.11:4579 , piece_id=JINRNEVUAP2VI5Y4CR7ALWDQIAIAFX3SP2VVZSXDDG6XPUKASE3Q, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.207967393 (58h26m7.079050438s) [noise=true , address=77.90.31.10:4550 , piece_id=PDCZ7VXNHQJ6LYPG64GZRYP4MRBGVPXCWTXSRT3UV2N6HGMDE2KA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.208416427 (58h26m7.079499472s) [noise=true , address=45.91.116.6:4003 , piece_id=CRONTV2B2AVZTLKU3YNIEGQZSON7XHHL7S3K6KNNJV4YR5FHQ3PA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.208830746 (58h26m7.079913791s) [noise=true , address=45.91.116.12:4254 , piece_id=UCKU4TBXJC4PSYHRDPUXETXLAILDOWA5NN7RCVVNYLVUBY7FBIFA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.209255245 (58h26m7.080338291s) [noise=true , address=145.79.227.16:4933 , piece_id=LHF64L3SWXFVZ5X5D67I44WSJNKRFB5G4YP7DFHDWOAYE23II3OA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.209662921 (58h26m7.080745966s) [noise=true , address=145.79.227.18:4010 , piece_id=S2EOTEV4TBNNDOBN7TIX76AJE2WI5P5DM3JK2NJB25TISYLS7VSQ, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.214493680 (58h26m7.085576725s) [noise=true , address=77.90.31.10:4539 , piece_id=27YFK55QLZDEULYABQ3OBDSPDDEIUJ2MJSD7X3A4K6HRFJMRQJUA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:08.555121654 (58h26m8.426204699s) [upload=51329]: Next returned nil segment Dec 8 01:55:08.555145519 (58h26m8.426228564s) [upload=51329]: Waiting for error group managing segments... Dec 8 01:55:08.935193756 (58h26m8.806276791s) [noise=true , address=64.49.13.27:4957 , piece_id=7METNUA4TAJ2FSYO5S4NXECVMMMCXKFL5CAHPQBQJ6WREQ3YFC3Q, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.093051314 (58h26m8.964134359s) [noise=true , address=145.79.227.16:4933 , piece_id=LHF64L3SWXFVZ5X5D67I44WSJNKRFB5G4YP7DFHDWOAYE23II3OA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.098812273 (58h26m8.969895318s) [noise=true , address=145.79.227.18:4010 , piece_id=S2EOTEV4TBNNDOBN7TIX76AJE2WI5P5DM3JK2NJB25TISYLS7VSQ, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.102616718 (58h26m8.973699763s) [noise=true , address=145.79.227.25:4627 , piece_id=K7UYU3W2CACUAQUUL72GG5F7DP5CHAHZ4M47TZG6NBN5Z6ZCDRBA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.118902949 (58h26m8.989985994s) [noise=true , address=145.79.227.34:4928 , piece_id=DRKF2DRSCBAYSZO66A2ZON2JS6HIJIT2YUWFZRS3W2CKFTV3NLVA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.124783838 (58h26m8.995866883s) [noise=true , address=77.90.31.18:4825 , piece_id=EJ4LO4TCEFB4OQQLFOEUYGQKK462ZPXXEIIY5CHF2PERXWDGQNDA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.126718282 (58h26m8.997801327s) [noise=true , address=145.79.227.11:4740 , piece_id=KL54LL2UQRT6MH2K3NQQJB3BKPX4QRWQMXRK27MW2U7CHFDCCWFA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.145158485 (58h26m9.01624153s) [noise=true , address=77.90.31.10:4550 , piece_id=PDCZ7VXNHQJ6LYPG64GZRYP4MRBGVPXCWTXSRT3UV2N6HGMDE2KA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.156650047 (58h26m9.027733092s) [noise=true , address=145.79.227.15:4908 , piece_id=JIUZSMOAKBFCFAX6RQ52TCNZMYBREQCYOEM75U3H2QYR4WH3KWCA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.162020363 (58h26m9.033103398s) [noise=true , address=77.90.31.10:4539 , piece_id=27YFK55QLZDEULYABQ3OBDSPDDEIUJ2MJSD7X3A4K6HRFJMRQJUA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.198899907 (58h26m9.069982942s) [noise=true , address=77.90.31.12:4621 , piece_id=S4FVJHEPIO2MVO56ZRIJ27RMTV723MERPLJLVD2T2GAO3T7K24TA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.208468337 (58h26m9.079551382s) [noise=true , address=45.91.116.4:4945 , piece_id=6UNWQQAWWWWC2XVUCDRCTAYJDTHBXAZJMXA6KLCTOP5PYSAJCCGA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.212217697 (58h26m9.083300732s) [noise=true , address=45.91.116.12:4254 , piece_id=UCKU4TBXJC4PSYHRDPUXETXLAILDOWA5NN7RCVVNYLVUBY7FBIFA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.215832390 (58h26m9.086915435s) [noise=true , address=77.90.31.11:4579 , piece_id=JINRNEVUAP2VI5Y4CR7ALWDQIAIAFX3SP2VVZSXDDG6XPUKASE3Q, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.215862086 (58h26m9.086945161s) [seg_pos={0 0}, upload=51329]: Segment reached optimal threshold of 14 pieces. Dec 8 01:55:09.216078098 (58h26m9.087161133s) [noise=true , address=45.91.116.6:4003 , piece_id=CRONTV2B2AVZTLKU3YNIEGQZSON7XHHL7S3K6KNNJV4YR5FHQ3PA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1veqoBGw6a8AGMoigyuNJMjcEvCT93xRq2B1d2KNhs6aEmSqVF): protocol: expected piece hash; context canceled; EOF Dec 8 01:55:09.216085662 (58h26m9.087168707s) [noise=true , address=45.91.116.6:4003 , piece_id=CRONTV2B2AVZTLKU3YNIEGQZSON7XHHL7S3K6KNNJV4YR5FHQ3PA, seg_pos={0 0}, upload=51329]: Long tail cancellation due to optimal threshold Dec 8 01:55:09.216307705 (58h26m9.08739075s) [noise=true , address=145.79.227.30:4786 , piece_id=VM65RUN3GUZZFUOHFYE7ZLPWU6COCQUMY3IK7GCNWPIUZAMEBR7Q, seg_pos={0 0}, upload=51329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125TWU7R4pkmrUxsNLhgTD475aLbjUJ7psA6drJtE7oDqZXYJ4Q): protocol: expected piece hash; context canceled; EOF Dec 8 01:55:09.216313956 (58h26m9.087396991s) [noise=true , address=145.79.227.30:4786 , piece_id=VM65RUN3GUZZFUOHFYE7ZLPWU6COCQUMY3IK7GCNWPIUZAMEBR7Q, seg_pos={0 0}, upload=51329]: Long tail cancellation due to optimal threshold Dec 8 01:55:09.216414809 (58h26m9.087497844s) [noise=true , address=77.90.31.16:4760 , piece_id=Q67LYTDJCAR2TFVSAIIYCUYM7D44JFPBLCSKRNLXH26EAW5UQ3GA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1e6p1TD5CqUyWdZjkrxh2kRqMuyDiMUr9eimfeayDAKwhtBrZ8): protocol: expected piece hash; context canceled; EOF Dec 8 01:55:09.216420681 (58h26m9.087503726s) [noise=true , address=77.90.31.16:4760 , piece_id=Q67LYTDJCAR2TFVSAIIYCUYM7D44JFPBLCSKRNLXH26EAW5UQ3GA, seg_pos={0 0}, upload=51329]: Long tail cancellation due to optimal threshold Dec 8 01:55:09.216562861 (58h26m9.087645906s) [noise=true , address=45.91.116.7:4037 , piece_id=UBNFDSXQBISC5QQSNGAAZIR7LRE7FCYRVYN5MMOY3A2ALEOQBYQQ, seg_pos={0 0}, upload=51329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iNPwE2XSfXcaXAXcrH2jcyNGYeDiiWvzAcDQffrmjPwTVqVmK): protocol: expected piece hash; context canceled; EOF Dec 8 01:55:09.216570025 (58h26m9.08765307s) [noise=true , address=45.91.116.7:4037 , piece_id=UBNFDSXQBISC5QQSNGAAZIR7LRE7FCYRVYN5MMOY3A2ALEOQBYQQ, seg_pos={0 0}, upload=51329]: Long tail cancellation due to optimal threshold Dec 8 01:55:09.216585564 (58h26m9.087668609s) [seg_pos={0 0}, upload=51329]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 8 01:55:09.216654365 (58h26m9.0877374s) [upload=51329]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 01:55:09.216659064 (58h26m9.087742099s) [upload=51329]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 01:55:35.852213564 (58h26m35.723296629s) [upload=51329]: Done uploading segments... Dec 8 01:55:35.852273619 (58h26m35.723356664s) [upload=51329]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.927311151s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 5.377135722s trace: https://jaeger.dev.storj.io/trace/7af096edea7ffcf5 FAIL (2025-12-08 02:21:39 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.377165578s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [..........................X......X................X....X....] duration: 3.411681945s trace: https://jaeger.dev.storj.io/trace/491130a80ad889c4 FAIL (2025-12-08 02:21:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.411710328s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 8.805816624s trace: https://jaeger.dev.storj.io/trace/27d92c196f473eec FAIL (2025-12-08 02:21:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 4891s) [upload=51469]: Got next segment. Inline: false Dec 8 02:04:21.143134547 (58h35m21.014217592s) [upload=51469]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 02:04:21.143141259 (58h35m21.014224304s) [upload=51469]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 02:04:22.267502706 (58h35m22.138585751s) [seg_pos={0 0}, upload=51469]: Begin upload segment... Dec 8 02:04:22.267624127 (58h35m22.138707172s) [seg_pos={0 0}, upload=51469]: Done begin upload segment. Dec 8 02:04:22.267698027 (58h35m22.138781073s) [noise=true , address=77.90.31.11:4554 , piece_id=BQP7XTRA2SBVINT4J7JAOF3ORNENBBIQN4ABMCEMQQ6FQL6QUOKA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.267712997 (58h35m22.138796042s) [noise=true , address=145.79.227.41:4087 , piece_id=XO6K3CXK7KZ5VEVXULHKQXLSSLIIWIX3VKQOSD7HY25WMNMCDLEQ, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.268434680 (58h35m22.139517715s) [noise=true , address=145.79.227.16:4910 , piece_id=ETE7ZBHT5IJUOBWCA5BGLKC2ANDEUB2M3DOMXWWS2RBWSJQBARRQ, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.268550331 (58h35m22.139633366s) [noise=true , address=145.79.227.26:4653 , piece_id=DKW72PEWPIOHPYHLI54PBU7NR3HGUBLFM3MN4XQQFWIIZ4FHIUNA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.268913111 (58h35m22.139996146s) [noise=true , address=145.79.227.23:4545 , piece_id=NBYCSU2KSHTAL223AHKHREQWKXUEQKJU46TY3SPYTKT5N5T2OUSA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.268924032 (58h35m22.140007067s) [noise=true , address=145.79.227.40:4051 , piece_id=7CQTGHFZ542WUHYX2CTOPN4PF2KDOOLDGV64THDBZCYOCXA3JWDA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.269398886 (58h35m22.140481921s) [noise=true , address=77.90.31.10:4518 , piece_id=6AR3NAFODP54SKG3SJSJSIQQFOYXO2C65WVNN4Z4XG426ZEOEDQA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.269414686 (58h35m22.140497721s) [noise=true , address=145.79.227.25:4619 , piece_id=XBNGVRIQLQUWKKBF4ULL7ZJDDT75QLY7ES2J7WHYWRD2ZLHOGU2Q, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.269753261 (58h35m22.140836296s) [noise=true , address=145.79.227.23:4547 , piece_id=5Y6V5QWLCWNOLT2GDNUEIRSG75KGNGIFLBCBGSIT7XH46RZYVDGQ, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.270016322 (58h35m22.141099367s) [noise=true , address=77.90.31.13:4628 , piece_id=LWTOCQRRA7HLI5ZPPRQ25IM22LMQNRR5H3TTOTSTGCFOYFZMABAA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.270259175 (58h35m22.14134222s) [noise=true , address=145.79.227.25:4617 , piece_id=S4ETCJ6WMCPXXQCYPPBBMHLMK2SVJV4S5VOKHGTDAGSGG6WRYH3Q, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.270655339 (58h35m22.141738374s) [noise=true , address=145.79.227.38:4977 , piece_id=73WXA33TWM4MFVTHDHW3PXJVABMCVMQYI3P42AHT3OYPKMXIVTUQ, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.271143158 (58h35m22.142246412s) [noise=true , address=145.79.227.28:4727 , piece_id=UYRK457JBF2AHCIXZ4XOIL7YHOVYZ7IPDUTB6WOZZUDEW5Q7NNCQ, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.271230565 (58h35m22.14231361s) [noise=true , address=145.79.227.39:4013 , piece_id=UI22RDXJ2DY3RFNSJVNT43XQJ6B2S3EO6SFLYHXW4VPVQJSYYF6Q, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.271662086 (58h35m22.142745121s) [noise=true , address=145.79.227.22:4511 , piece_id=GGA463DAOSZCW7SQGZW2LGFRQVNUOUJ2L2GXRXLSDPC65KVPY3TA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.272592658 (58h35m22.143675703s) [noise=true , address=145.79.227.24:4583 , piece_id=WYTXGXG77ZW4RGWX7YWDU2NH6R5CCPVBPQH3ISHUAJVKZJRONS5Q, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:24.009199123 (58h35m23.880282158s) [upload=51469]: Next returned nil segment Dec 8 02:04:24.009221235 (58h35m23.88030428s) [upload=51469]: Waiting for error group managing segments... Dec 8 02:04:24.293906896 (58h35m24.164989931s) [noise=true , address=145.79.227.24:4583 , piece_id=WYTXGXG77ZW4RGWX7YWDU2NH6R5CCPVBPQH3ISHUAJVKZJRONS5Q, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.298576738 (58h35m24.169659783s) [noise=true , address=145.79.227.28:4727 , piece_id=UYRK457JBF2AHCIXZ4XOIL7YHOVYZ7IPDUTB6WOZZUDEW5Q7NNCQ, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.301356571 (58h35m24.172439616s) [noise=true , address=145.79.227.25:4619 , piece_id=XBNGVRIQLQUWKKBF4ULL7ZJDDT75QLY7ES2J7WHYWRD2ZLHOGU2Q, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.326466765 (58h35m24.19754981s) [noise=true , address=145.79.227.16:4910 , piece_id=ETE7ZBHT5IJUOBWCA5BGLKC2ANDEUB2M3DOMXWWS2RBWSJQBARRQ, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.328633530 (58h35m24.199716586s) [noise=true , address=145.79.227.26:4653 , piece_id=DKW72PEWPIOHPYHLI54PBU7NR3HGUBLFM3MN4XQQFWIIZ4FHIUNA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.360508106 (58h35m24.231591151s) [noise=true , address=145.79.227.40:4051 , piece_id=7CQTGHFZ542WUHYX2CTOPN4PF2KDOOLDGV64THDBZCYOCXA3JWDA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.361116263 (58h35m24.232199308s) [noise=true , address=145.79.227.25:4617 , piece_id=S4ETCJ6WMCPXXQCYPPBBMHLMK2SVJV4S5VOKHGTDAGSGG6WRYH3Q, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.361695086 (58h35m24.232778131s) [noise=true , address=145.79.227.41:4087 , piece_id=XO6K3CXK7KZ5VEVXULHKQXLSSLIIWIX3VKQOSD7HY25WMNMCDLEQ, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.371520525 (58h35m24.24260356s) [noise=true , address=145.79.227.38:4977 , piece_id=73WXA33TWM4MFVTHDHW3PXJVABMCVMQYI3P42AHT3OYPKMXIVTUQ, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.381271893 (58h35m24.252354928s) [noise=true , address=145.79.227.23:4545 , piece_id=NBYCSU2KSHTAL223AHKHREQWKXUEQKJU46TY3SPYTKT5N5T2OUSA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.386403053 (58h35m24.257486098s) [noise=true , address=77.90.31.11:4554 , piece_id=BQP7XTRA2SBVINT4J7JAOF3ORNENBBIQN4ABMCEMQQ6FQL6QUOKA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.386427209 (58h35m24.257510254s) [noise=true , address=77.90.31.10:4518 , piece_id=6AR3NAFODP54SKG3SJSJSIQQFOYXO2C65WVNN4Z4XG426ZEOEDQA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.400489386 (58h35m24.271572431s) [noise=true , address=77.90.31.13:4628 , piece_id=LWTOCQRRA7HLI5ZPPRQ25IM22LMQNRR5H3TTOTSTGCFOYFZMABAA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.400513903 (58h35m24.271596938s) [seg_pos={0 0}, upload=51469]: Segment reached optimal threshold of 13 pieces. Dec 8 02:04:24.400813233 (58h35m24.271896288s) [noise=true , address=145.79.227.39:4013 , piece_id=UI22RDXJ2DY3RFNSJVNT43XQJ6B2S3EO6SFLYHXW4VPVQJSYYF6Q, seg_pos={0 0}, upload=51469]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b): protocol: write tcp 188.166.225.170:39434->145.79.227.39:4013: use of closed network connection; write tcp 188.166.225.170:39434->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39434->145.79.227.39:4013: use of closed network connection Dec 8 02:04:24.400825677 (58h35m24.271908722s) [noise=true , address=145.79.227.39:4013 , piece_id=UI22RDXJ2DY3RFNSJVNT43XQJ6B2S3EO6SFLYHXW4VPVQJSYYF6Q, seg_pos={0 0}, upload=51469]: Long tail cancellation due to optimal threshold Dec 8 02:04:24.400873027 (58h35m24.271956082s) [noise=true , address=145.79.227.22:4511 , piece_id=GGA463DAOSZCW7SQGZW2LGFRQVNUOUJ2L2GXRXLSDPC65KVPY3TA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG): protocol: write tcp 188.166.225.170:47926->145.79.227.22:4511: use of closed network connection; write tcp 188.166.225.170:47926->145.79.227.22:4511: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47926->145.79.227.22:4511: use of closed network connection Dec 8 02:04:24.400894248 (58h35m24.271977293s) [noise=true , address=145.79.227.22:4511 , piece_id=GGA463DAOSZCW7SQGZW2LGFRQVNUOUJ2L2GXRXLSDPC65KVPY3TA, seg_pos={0 0}, upload=51469]: Long tail cancellation due to optimal threshold Dec 8 02:04:24.401247129 (58h35m24.272330184s) [noise=true , address=145.79.227.23:4547 , piece_id=5Y6V5QWLCWNOLT2GDNUEIRSG75KGNGIFLBCBGSIT7XH46RZYVDGQ, seg_pos={0 0}, upload=51469]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW): protocol: read tcp 188.166.225.170:43344->145.79.227.23:4547: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 02:04:24.401260003 (58h35m24.272343059s) [noise=true , address=145.79.227.23:4547 , piece_id=5Y6V5QWLCWNOLT2GDNUEIRSG75KGNGIFLBCBGSIT7XH46RZYVDGQ, seg_pos={0 0}, upload=51469]: Long tail cancellation due to optimal threshold Dec 8 02:04:24.401285452 (58h35m24.272368497s) [seg_pos={0 0}, upload=51469]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 02:04:24.401382247 (58h35m24.272465292s) [upload=51469]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 02:04:24.401391394 (58h35m24.272474429s) [upload=51469]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 02:05:39.049922629 (58h36m38.921005704s) [upload=51469]: Done uploading segments... Dec 8 02:05:39.050003342 (58h36m38.921086377s) [upload=51469]: Upload finished. err: metaclient: context deadline exceeded Duration: 8.80586786s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 5.563995039s trace: https://jaeger.dev.storj.io/trace/57258282a8e974fd FAIL (2025-12-08 02:21:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== UA4TAJ2FSYO5S4NXECVMMMCXKFL5CAHPQBQJ6WREQ3YFC3Q, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.202906305 (58h26m7.07398934s) [noise=true , address=45.91.116.4:4945 , piece_id=6UNWQQAWWWWC2XVUCDRCTAYJDTHBXAZJMXA6KLCTOP5PYSAJCCGA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.203371201 (58h26m7.074454246s) [noise=true , address=145.79.227.11:4740 , piece_id=KL54LL2UQRT6MH2K3NQQJB3BKPX4QRWQMXRK27MW2U7CHFDCCWFA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.203861354 (58h26m7.07494439s) [noise=true , address=77.90.31.12:4621 , piece_id=S4FVJHEPIO2MVO56ZRIJ27RMTV723MERPLJLVD2T2GAO3T7K24TA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.204290401 (58h26m7.075373446s) [noise=true , address=145.79.227.34:4928 , piece_id=DRKF2DRSCBAYSZO66A2ZON2JS6HIJIT2YUWFZRS3W2CKFTV3NLVA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.204711363 (58h26m7.075794398s) [noise=true , address=145.79.227.15:4908 , piece_id=JIUZSMOAKBFCFAX6RQ52TCNZMYBREQCYOEM75U3H2QYR4WH3KWCA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.205118208 (58h26m7.076201253s) [noise=true , address=77.90.31.18:4825 , piece_id=EJ4LO4TCEFB4OQQLFOEUYGQKK462ZPXXEIIY5CHF2PERXWDGQNDA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.205554277 (58h26m7.076637323s) [noise=true , address=145.79.227.30:4786 , piece_id=VM65RUN3GUZZFUOHFYE7ZLPWU6COCQUMY3IK7GCNWPIUZAMEBR7Q, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.206116137 (58h26m7.077199182s) [noise=true , address=145.79.227.25:4627 , piece_id=K7UYU3W2CACUAQUUL72GG5F7DP5CHAHZ4M47TZG6NBN5Z6ZCDRBA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.206664562 (58h26m7.077747607s) [noise=true , address=77.90.31.16:4760 , piece_id=Q67LYTDJCAR2TFVSAIIYCUYM7D44JFPBLCSKRNLXH26EAW5UQ3GA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.207102556 (58h26m7.078185601s) [noise=true , address=45.91.116.7:4037 , piece_id=UBNFDSXQBISC5QQSNGAAZIR7LRE7FCYRVYN5MMOY3A2ALEOQBYQQ, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.207519911 (58h26m7.078602956s) [noise=true , address=77.90.31.11:4579 , piece_id=JINRNEVUAP2VI5Y4CR7ALWDQIAIAFX3SP2VVZSXDDG6XPUKASE3Q, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.207967393 (58h26m7.079050438s) [noise=true , address=77.90.31.10:4550 , piece_id=PDCZ7VXNHQJ6LYPG64GZRYP4MRBGVPXCWTXSRT3UV2N6HGMDE2KA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.208416427 (58h26m7.079499472s) [noise=true , address=45.91.116.6:4003 , piece_id=CRONTV2B2AVZTLKU3YNIEGQZSON7XHHL7S3K6KNNJV4YR5FHQ3PA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.208830746 (58h26m7.079913791s) [noise=true , address=45.91.116.12:4254 , piece_id=UCKU4TBXJC4PSYHRDPUXETXLAILDOWA5NN7RCVVNYLVUBY7FBIFA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.209255245 (58h26m7.080338291s) [noise=true , address=145.79.227.16:4933 , piece_id=LHF64L3SWXFVZ5X5D67I44WSJNKRFB5G4YP7DFHDWOAYE23II3OA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.209662921 (58h26m7.080745966s) [noise=true , address=145.79.227.18:4010 , piece_id=S2EOTEV4TBNNDOBN7TIX76AJE2WI5P5DM3JK2NJB25TISYLS7VSQ, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.214493680 (58h26m7.085576725s) [noise=true , address=77.90.31.10:4539 , piece_id=27YFK55QLZDEULYABQ3OBDSPDDEIUJ2MJSD7X3A4K6HRFJMRQJUA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:08.555121654 (58h26m8.426204699s) [upload=51329]: Next returned nil segment Dec 8 01:55:08.555145519 (58h26m8.426228564s) [upload=51329]: Waiting for error group managing segments... Dec 8 01:55:08.935193756 (58h26m8.806276791s) [noise=true , address=64.49.13.27:4957 , piece_id=7METNUA4TAJ2FSYO5S4NXECVMMMCXKFL5CAHPQBQJ6WREQ3YFC3Q, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.093051314 (58h26m8.964134359s) [noise=true , address=145.79.227.16:4933 , piece_id=LHF64L3SWXFVZ5X5D67I44WSJNKRFB5G4YP7DFHDWOAYE23II3OA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.098812273 (58h26m8.969895318s) [noise=true , address=145.79.227.18:4010 , piece_id=S2EOTEV4TBNNDOBN7TIX76AJE2WI5P5DM3JK2NJB25TISYLS7VSQ, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.102616718 (58h26m8.973699763s) [noise=true , address=145.79.227.25:4627 , piece_id=K7UYU3W2CACUAQUUL72GG5F7DP5CHAHZ4M47TZG6NBN5Z6ZCDRBA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.118902949 (58h26m8.989985994s) [noise=true , address=145.79.227.34:4928 , piece_id=DRKF2DRSCBAYSZO66A2ZON2JS6HIJIT2YUWFZRS3W2CKFTV3NLVA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.124783838 (58h26m8.995866883s) [noise=true , address=77.90.31.18:4825 , piece_id=EJ4LO4TCEFB4OQQLFOEUYGQKK462ZPXXEIIY5CHF2PERXWDGQNDA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.126718282 (58h26m8.997801327s) [noise=true , address=145.79.227.11:4740 , piece_id=KL54LL2UQRT6MH2K3NQQJB3BKPX4QRWQMXRK27MW2U7CHFDCCWFA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.145158485 (58h26m9.01624153s) [noise=true , address=77.90.31.10:4550 , piece_id=PDCZ7VXNHQJ6LYPG64GZRYP4MRBGVPXCWTXSRT3UV2N6HGMDE2KA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.156650047 (58h26m9.027733092s) [noise=true , address=145.79.227.15:4908 , piece_id=JIUZSMOAKBFCFAX6RQ52TCNZMYBREQCYOEM75U3H2QYR4WH3KWCA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.162020363 (58h26m9.033103398s) [noise=true , address=77.90.31.10:4539 , piece_id=27YFK55QLZDEULYABQ3OBDSPDDEIUJ2MJSD7X3A4K6HRFJMRQJUA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.198899907 (58h26m9.069982942s) [noise=true , address=77.90.31.12:4621 , piece_id=S4FVJHEPIO2MVO56ZRIJ27RMTV723MERPLJLVD2T2GAO3T7K24TA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.208468337 (58h26m9.079551382s) [noise=true , address=45.91.116.4:4945 , piece_id=6UNWQQAWWWWC2XVUCDRCTAYJDTHBXAZJMXA6KLCTOP5PYSAJCCGA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.212217697 (58h26m9.083300732s) [noise=true , address=45.91.116.12:4254 , piece_id=UCKU4TBXJC4PSYHRDPUXETXLAILDOWA5NN7RCVVNYLVUBY7FBIFA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.215832390 (58h26m9.086915435s) [noise=true , address=77.90.31.11:4579 , piece_id=JINRNEVUAP2VI5Y4CR7ALWDQIAIAFX3SP2VVZSXDDG6XPUKASE3Q, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.215862086 (58h26m9.086945161s) [seg_pos={0 0}, upload=51329]: Segment reached optimal threshold of 14 pieces. Dec 8 01:55:09.216078098 (58h26m9.087161133s) [noise=true , address=45.91.116.6:4003 , piece_id=CRONTV2B2AVZTLKU3YNIEGQZSON7XHHL7S3K6KNNJV4YR5FHQ3PA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1veqoBGw6a8AGMoigyuNJMjcEvCT93xRq2B1d2KNhs6aEmSqVF): protocol: expected piece hash; context canceled; EOF Dec 8 01:55:09.216085662 (58h26m9.087168707s) [noise=true , address=45.91.116.6:4003 , piece_id=CRONTV2B2AVZTLKU3YNIEGQZSON7XHHL7S3K6KNNJV4YR5FHQ3PA, seg_pos={0 0}, upload=51329]: Long tail cancellation due to optimal threshold Dec 8 01:55:09.216307705 (58h26m9.08739075s) [noise=true , address=145.79.227.30:4786 , piece_id=VM65RUN3GUZZFUOHFYE7ZLPWU6COCQUMY3IK7GCNWPIUZAMEBR7Q, seg_pos={0 0}, upload=51329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125TWU7R4pkmrUxsNLhgTD475aLbjUJ7psA6drJtE7oDqZXYJ4Q): protocol: expected piece hash; context canceled; EOF Dec 8 01:55:09.216313956 (58h26m9.087396991s) [noise=true , address=145.79.227.30:4786 , piece_id=VM65RUN3GUZZFUOHFYE7ZLPWU6COCQUMY3IK7GCNWPIUZAMEBR7Q, seg_pos={0 0}, upload=51329]: Long tail cancellation due to optimal threshold Dec 8 01:55:09.216414809 (58h26m9.087497844s) [noise=true , address=77.90.31.16:4760 , piece_id=Q67LYTDJCAR2TFVSAIIYCUYM7D44JFPBLCSKRNLXH26EAW5UQ3GA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1e6p1TD5CqUyWdZjkrxh2kRqMuyDiMUr9eimfeayDAKwhtBrZ8): protocol: expected piece hash; context canceled; EOF Dec 8 01:55:09.216420681 (58h26m9.087503726s) [noise=true , address=77.90.31.16:4760 , piece_id=Q67LYTDJCAR2TFVSAIIYCUYM7D44JFPBLCSKRNLXH26EAW5UQ3GA, seg_pos={0 0}, upload=51329]: Long tail cancellation due to optimal threshold Dec 8 01:55:09.216562861 (58h26m9.087645906s) [noise=true , address=45.91.116.7:4037 , piece_id=UBNFDSXQBISC5QQSNGAAZIR7LRE7FCYRVYN5MMOY3A2ALEOQBYQQ, seg_pos={0 0}, upload=51329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iNPwE2XSfXcaXAXcrH2jcyNGYeDiiWvzAcDQffrmjPwTVqVmK): protocol: expected piece hash; context canceled; EOF Dec 8 01:55:09.216570025 (58h26m9.08765307s) [noise=true , address=45.91.116.7:4037 , piece_id=UBNFDSXQBISC5QQSNGAAZIR7LRE7FCYRVYN5MMOY3A2ALEOQBYQQ, seg_pos={0 0}, upload=51329]: Long tail cancellation due to optimal threshold Dec 8 01:55:09.216585564 (58h26m9.087668609s) [seg_pos={0 0}, upload=51329]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 8 01:55:09.216654365 (58h26m9.0877374s) [upload=51329]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 01:55:09.216659064 (58h26m9.087742099s) [upload=51329]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 01:55:35.852213564 (58h26m35.723296629s) [upload=51329]: Done uploading segments... Dec 8 01:55:35.852273619 (58h26m35.723356664s) [upload=51329]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.564023955s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 3 out of 60 is failed (threshold=3) [........X................X.........X........................] duration: 6.187169169s trace: https://jaeger.dev.storj.io/trace/e7164007ac0fd55 FAIL (2025-12-08 02:21:14 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 6.187209135s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 7.71784897s trace: https://jaeger.dev.storj.io/trace/27d92c196f473eec FAIL (2025-12-08 02:20:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 4891s) [upload=51469]: Got next segment. Inline: false Dec 8 02:04:21.143134547 (58h35m21.014217592s) [upload=51469]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 02:04:21.143141259 (58h35m21.014224304s) [upload=51469]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 02:04:22.267502706 (58h35m22.138585751s) [seg_pos={0 0}, upload=51469]: Begin upload segment... Dec 8 02:04:22.267624127 (58h35m22.138707172s) [seg_pos={0 0}, upload=51469]: Done begin upload segment. Dec 8 02:04:22.267698027 (58h35m22.138781073s) [noise=true , address=77.90.31.11:4554 , piece_id=BQP7XTRA2SBVINT4J7JAOF3ORNENBBIQN4ABMCEMQQ6FQL6QUOKA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.267712997 (58h35m22.138796042s) [noise=true , address=145.79.227.41:4087 , piece_id=XO6K3CXK7KZ5VEVXULHKQXLSSLIIWIX3VKQOSD7HY25WMNMCDLEQ, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.268434680 (58h35m22.139517715s) [noise=true , address=145.79.227.16:4910 , piece_id=ETE7ZBHT5IJUOBWCA5BGLKC2ANDEUB2M3DOMXWWS2RBWSJQBARRQ, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.268550331 (58h35m22.139633366s) [noise=true , address=145.79.227.26:4653 , piece_id=DKW72PEWPIOHPYHLI54PBU7NR3HGUBLFM3MN4XQQFWIIZ4FHIUNA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.268913111 (58h35m22.139996146s) [noise=true , address=145.79.227.23:4545 , piece_id=NBYCSU2KSHTAL223AHKHREQWKXUEQKJU46TY3SPYTKT5N5T2OUSA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.268924032 (58h35m22.140007067s) [noise=true , address=145.79.227.40:4051 , piece_id=7CQTGHFZ542WUHYX2CTOPN4PF2KDOOLDGV64THDBZCYOCXA3JWDA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.269398886 (58h35m22.140481921s) [noise=true , address=77.90.31.10:4518 , piece_id=6AR3NAFODP54SKG3SJSJSIQQFOYXO2C65WVNN4Z4XG426ZEOEDQA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.269414686 (58h35m22.140497721s) [noise=true , address=145.79.227.25:4619 , piece_id=XBNGVRIQLQUWKKBF4ULL7ZJDDT75QLY7ES2J7WHYWRD2ZLHOGU2Q, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.269753261 (58h35m22.140836296s) [noise=true , address=145.79.227.23:4547 , piece_id=5Y6V5QWLCWNOLT2GDNUEIRSG75KGNGIFLBCBGSIT7XH46RZYVDGQ, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.270016322 (58h35m22.141099367s) [noise=true , address=77.90.31.13:4628 , piece_id=LWTOCQRRA7HLI5ZPPRQ25IM22LMQNRR5H3TTOTSTGCFOYFZMABAA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.270259175 (58h35m22.14134222s) [noise=true , address=145.79.227.25:4617 , piece_id=S4ETCJ6WMCPXXQCYPPBBMHLMK2SVJV4S5VOKHGTDAGSGG6WRYH3Q, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.270655339 (58h35m22.141738374s) [noise=true , address=145.79.227.38:4977 , piece_id=73WXA33TWM4MFVTHDHW3PXJVABMCVMQYI3P42AHT3OYPKMXIVTUQ, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.271143158 (58h35m22.142246412s) [noise=true , address=145.79.227.28:4727 , piece_id=UYRK457JBF2AHCIXZ4XOIL7YHOVYZ7IPDUTB6WOZZUDEW5Q7NNCQ, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.271230565 (58h35m22.14231361s) [noise=true , address=145.79.227.39:4013 , piece_id=UI22RDXJ2DY3RFNSJVNT43XQJ6B2S3EO6SFLYHXW4VPVQJSYYF6Q, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.271662086 (58h35m22.142745121s) [noise=true , address=145.79.227.22:4511 , piece_id=GGA463DAOSZCW7SQGZW2LGFRQVNUOUJ2L2GXRXLSDPC65KVPY3TA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.272592658 (58h35m22.143675703s) [noise=true , address=145.79.227.24:4583 , piece_id=WYTXGXG77ZW4RGWX7YWDU2NH6R5CCPVBPQH3ISHUAJVKZJRONS5Q, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:24.009199123 (58h35m23.880282158s) [upload=51469]: Next returned nil segment Dec 8 02:04:24.009221235 (58h35m23.88030428s) [upload=51469]: Waiting for error group managing segments... Dec 8 02:04:24.293906896 (58h35m24.164989931s) [noise=true , address=145.79.227.24:4583 , piece_id=WYTXGXG77ZW4RGWX7YWDU2NH6R5CCPVBPQH3ISHUAJVKZJRONS5Q, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.298576738 (58h35m24.169659783s) [noise=true , address=145.79.227.28:4727 , piece_id=UYRK457JBF2AHCIXZ4XOIL7YHOVYZ7IPDUTB6WOZZUDEW5Q7NNCQ, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.301356571 (58h35m24.172439616s) [noise=true , address=145.79.227.25:4619 , piece_id=XBNGVRIQLQUWKKBF4ULL7ZJDDT75QLY7ES2J7WHYWRD2ZLHOGU2Q, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.326466765 (58h35m24.19754981s) [noise=true , address=145.79.227.16:4910 , piece_id=ETE7ZBHT5IJUOBWCA5BGLKC2ANDEUB2M3DOMXWWS2RBWSJQBARRQ, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.328633530 (58h35m24.199716586s) [noise=true , address=145.79.227.26:4653 , piece_id=DKW72PEWPIOHPYHLI54PBU7NR3HGUBLFM3MN4XQQFWIIZ4FHIUNA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.360508106 (58h35m24.231591151s) [noise=true , address=145.79.227.40:4051 , piece_id=7CQTGHFZ542WUHYX2CTOPN4PF2KDOOLDGV64THDBZCYOCXA3JWDA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.361116263 (58h35m24.232199308s) [noise=true , address=145.79.227.25:4617 , piece_id=S4ETCJ6WMCPXXQCYPPBBMHLMK2SVJV4S5VOKHGTDAGSGG6WRYH3Q, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.361695086 (58h35m24.232778131s) [noise=true , address=145.79.227.41:4087 , piece_id=XO6K3CXK7KZ5VEVXULHKQXLSSLIIWIX3VKQOSD7HY25WMNMCDLEQ, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.371520525 (58h35m24.24260356s) [noise=true , address=145.79.227.38:4977 , piece_id=73WXA33TWM4MFVTHDHW3PXJVABMCVMQYI3P42AHT3OYPKMXIVTUQ, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.381271893 (58h35m24.252354928s) [noise=true , address=145.79.227.23:4545 , piece_id=NBYCSU2KSHTAL223AHKHREQWKXUEQKJU46TY3SPYTKT5N5T2OUSA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.386403053 (58h35m24.257486098s) [noise=true , address=77.90.31.11:4554 , piece_id=BQP7XTRA2SBVINT4J7JAOF3ORNENBBIQN4ABMCEMQQ6FQL6QUOKA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.386427209 (58h35m24.257510254s) [noise=true , address=77.90.31.10:4518 , piece_id=6AR3NAFODP54SKG3SJSJSIQQFOYXO2C65WVNN4Z4XG426ZEOEDQA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.400489386 (58h35m24.271572431s) [noise=true , address=77.90.31.13:4628 , piece_id=LWTOCQRRA7HLI5ZPPRQ25IM22LMQNRR5H3TTOTSTGCFOYFZMABAA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.400513903 (58h35m24.271596938s) [seg_pos={0 0}, upload=51469]: Segment reached optimal threshold of 13 pieces. Dec 8 02:04:24.400813233 (58h35m24.271896288s) [noise=true , address=145.79.227.39:4013 , piece_id=UI22RDXJ2DY3RFNSJVNT43XQJ6B2S3EO6SFLYHXW4VPVQJSYYF6Q, seg_pos={0 0}, upload=51469]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b): protocol: write tcp 188.166.225.170:39434->145.79.227.39:4013: use of closed network connection; write tcp 188.166.225.170:39434->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39434->145.79.227.39:4013: use of closed network connection Dec 8 02:04:24.400825677 (58h35m24.271908722s) [noise=true , address=145.79.227.39:4013 , piece_id=UI22RDXJ2DY3RFNSJVNT43XQJ6B2S3EO6SFLYHXW4VPVQJSYYF6Q, seg_pos={0 0}, upload=51469]: Long tail cancellation due to optimal threshold Dec 8 02:04:24.400873027 (58h35m24.271956082s) [noise=true , address=145.79.227.22:4511 , piece_id=GGA463DAOSZCW7SQGZW2LGFRQVNUOUJ2L2GXRXLSDPC65KVPY3TA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG): protocol: write tcp 188.166.225.170:47926->145.79.227.22:4511: use of closed network connection; write tcp 188.166.225.170:47926->145.79.227.22:4511: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47926->145.79.227.22:4511: use of closed network connection Dec 8 02:04:24.400894248 (58h35m24.271977293s) [noise=true , address=145.79.227.22:4511 , piece_id=GGA463DAOSZCW7SQGZW2LGFRQVNUOUJ2L2GXRXLSDPC65KVPY3TA, seg_pos={0 0}, upload=51469]: Long tail cancellation due to optimal threshold Dec 8 02:04:24.401247129 (58h35m24.272330184s) [noise=true , address=145.79.227.23:4547 , piece_id=5Y6V5QWLCWNOLT2GDNUEIRSG75KGNGIFLBCBGSIT7XH46RZYVDGQ, seg_pos={0 0}, upload=51469]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW): protocol: read tcp 188.166.225.170:43344->145.79.227.23:4547: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 02:04:24.401260003 (58h35m24.272343059s) [noise=true , address=145.79.227.23:4547 , piece_id=5Y6V5QWLCWNOLT2GDNUEIRSG75KGNGIFLBCBGSIT7XH46RZYVDGQ, seg_pos={0 0}, upload=51469]: Long tail cancellation due to optimal threshold Dec 8 02:04:24.401285452 (58h35m24.272368497s) [seg_pos={0 0}, upload=51469]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 02:04:24.401382247 (58h35m24.272465292s) [upload=51469]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 02:04:24.401391394 (58h35m24.272474429s) [upload=51469]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 02:05:39.049922629 (58h36m38.921005704s) [upload=51469]: Done uploading segments... Dec 8 02:05:39.050003342 (58h36m38.921086377s) [upload=51469]: Upload finished. err: metaclient: context deadline exceeded Duration: 7.717881573s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 5.751274354s trace: https://jaeger.dev.storj.io/trace/57258282a8e974fd FAIL (2025-12-08 02:20:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== UA4TAJ2FSYO5S4NXECVMMMCXKFL5CAHPQBQJ6WREQ3YFC3Q, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.202906305 (58h26m7.07398934s) [noise=true , address=45.91.116.4:4945 , piece_id=6UNWQQAWWWWC2XVUCDRCTAYJDTHBXAZJMXA6KLCTOP5PYSAJCCGA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.203371201 (58h26m7.074454246s) [noise=true , address=145.79.227.11:4740 , piece_id=KL54LL2UQRT6MH2K3NQQJB3BKPX4QRWQMXRK27MW2U7CHFDCCWFA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.203861354 (58h26m7.07494439s) [noise=true , address=77.90.31.12:4621 , piece_id=S4FVJHEPIO2MVO56ZRIJ27RMTV723MERPLJLVD2T2GAO3T7K24TA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.204290401 (58h26m7.075373446s) [noise=true , address=145.79.227.34:4928 , piece_id=DRKF2DRSCBAYSZO66A2ZON2JS6HIJIT2YUWFZRS3W2CKFTV3NLVA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.204711363 (58h26m7.075794398s) [noise=true , address=145.79.227.15:4908 , piece_id=JIUZSMOAKBFCFAX6RQ52TCNZMYBREQCYOEM75U3H2QYR4WH3KWCA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.205118208 (58h26m7.076201253s) [noise=true , address=77.90.31.18:4825 , piece_id=EJ4LO4TCEFB4OQQLFOEUYGQKK462ZPXXEIIY5CHF2PERXWDGQNDA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.205554277 (58h26m7.076637323s) [noise=true , address=145.79.227.30:4786 , piece_id=VM65RUN3GUZZFUOHFYE7ZLPWU6COCQUMY3IK7GCNWPIUZAMEBR7Q, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.206116137 (58h26m7.077199182s) [noise=true , address=145.79.227.25:4627 , piece_id=K7UYU3W2CACUAQUUL72GG5F7DP5CHAHZ4M47TZG6NBN5Z6ZCDRBA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.206664562 (58h26m7.077747607s) [noise=true , address=77.90.31.16:4760 , piece_id=Q67LYTDJCAR2TFVSAIIYCUYM7D44JFPBLCSKRNLXH26EAW5UQ3GA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.207102556 (58h26m7.078185601s) [noise=true , address=45.91.116.7:4037 , piece_id=UBNFDSXQBISC5QQSNGAAZIR7LRE7FCYRVYN5MMOY3A2ALEOQBYQQ, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.207519911 (58h26m7.078602956s) [noise=true , address=77.90.31.11:4579 , piece_id=JINRNEVUAP2VI5Y4CR7ALWDQIAIAFX3SP2VVZSXDDG6XPUKASE3Q, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.207967393 (58h26m7.079050438s) [noise=true , address=77.90.31.10:4550 , piece_id=PDCZ7VXNHQJ6LYPG64GZRYP4MRBGVPXCWTXSRT3UV2N6HGMDE2KA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.208416427 (58h26m7.079499472s) [noise=true , address=45.91.116.6:4003 , piece_id=CRONTV2B2AVZTLKU3YNIEGQZSON7XHHL7S3K6KNNJV4YR5FHQ3PA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.208830746 (58h26m7.079913791s) [noise=true , address=45.91.116.12:4254 , piece_id=UCKU4TBXJC4PSYHRDPUXETXLAILDOWA5NN7RCVVNYLVUBY7FBIFA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.209255245 (58h26m7.080338291s) [noise=true , address=145.79.227.16:4933 , piece_id=LHF64L3SWXFVZ5X5D67I44WSJNKRFB5G4YP7DFHDWOAYE23II3OA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.209662921 (58h26m7.080745966s) [noise=true , address=145.79.227.18:4010 , piece_id=S2EOTEV4TBNNDOBN7TIX76AJE2WI5P5DM3JK2NJB25TISYLS7VSQ, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.214493680 (58h26m7.085576725s) [noise=true , address=77.90.31.10:4539 , piece_id=27YFK55QLZDEULYABQ3OBDSPDDEIUJ2MJSD7X3A4K6HRFJMRQJUA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:08.555121654 (58h26m8.426204699s) [upload=51329]: Next returned nil segment Dec 8 01:55:08.555145519 (58h26m8.426228564s) [upload=51329]: Waiting for error group managing segments... Dec 8 01:55:08.935193756 (58h26m8.806276791s) [noise=true , address=64.49.13.27:4957 , piece_id=7METNUA4TAJ2FSYO5S4NXECVMMMCXKFL5CAHPQBQJ6WREQ3YFC3Q, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.093051314 (58h26m8.964134359s) [noise=true , address=145.79.227.16:4933 , piece_id=LHF64L3SWXFVZ5X5D67I44WSJNKRFB5G4YP7DFHDWOAYE23II3OA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.098812273 (58h26m8.969895318s) [noise=true , address=145.79.227.18:4010 , piece_id=S2EOTEV4TBNNDOBN7TIX76AJE2WI5P5DM3JK2NJB25TISYLS7VSQ, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.102616718 (58h26m8.973699763s) [noise=true , address=145.79.227.25:4627 , piece_id=K7UYU3W2CACUAQUUL72GG5F7DP5CHAHZ4M47TZG6NBN5Z6ZCDRBA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.118902949 (58h26m8.989985994s) [noise=true , address=145.79.227.34:4928 , piece_id=DRKF2DRSCBAYSZO66A2ZON2JS6HIJIT2YUWFZRS3W2CKFTV3NLVA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.124783838 (58h26m8.995866883s) [noise=true , address=77.90.31.18:4825 , piece_id=EJ4LO4TCEFB4OQQLFOEUYGQKK462ZPXXEIIY5CHF2PERXWDGQNDA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.126718282 (58h26m8.997801327s) [noise=true , address=145.79.227.11:4740 , piece_id=KL54LL2UQRT6MH2K3NQQJB3BKPX4QRWQMXRK27MW2U7CHFDCCWFA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.145158485 (58h26m9.01624153s) [noise=true , address=77.90.31.10:4550 , piece_id=PDCZ7VXNHQJ6LYPG64GZRYP4MRBGVPXCWTXSRT3UV2N6HGMDE2KA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.156650047 (58h26m9.027733092s) [noise=true , address=145.79.227.15:4908 , piece_id=JIUZSMOAKBFCFAX6RQ52TCNZMYBREQCYOEM75U3H2QYR4WH3KWCA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.162020363 (58h26m9.033103398s) [noise=true , address=77.90.31.10:4539 , piece_id=27YFK55QLZDEULYABQ3OBDSPDDEIUJ2MJSD7X3A4K6HRFJMRQJUA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.198899907 (58h26m9.069982942s) [noise=true , address=77.90.31.12:4621 , piece_id=S4FVJHEPIO2MVO56ZRIJ27RMTV723MERPLJLVD2T2GAO3T7K24TA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.208468337 (58h26m9.079551382s) [noise=true , address=45.91.116.4:4945 , piece_id=6UNWQQAWWWWC2XVUCDRCTAYJDTHBXAZJMXA6KLCTOP5PYSAJCCGA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.212217697 (58h26m9.083300732s) [noise=true , address=45.91.116.12:4254 , piece_id=UCKU4TBXJC4PSYHRDPUXETXLAILDOWA5NN7RCVVNYLVUBY7FBIFA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.215832390 (58h26m9.086915435s) [noise=true , address=77.90.31.11:4579 , piece_id=JINRNEVUAP2VI5Y4CR7ALWDQIAIAFX3SP2VVZSXDDG6XPUKASE3Q, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.215862086 (58h26m9.086945161s) [seg_pos={0 0}, upload=51329]: Segment reached optimal threshold of 14 pieces. Dec 8 01:55:09.216078098 (58h26m9.087161133s) [noise=true , address=45.91.116.6:4003 , piece_id=CRONTV2B2AVZTLKU3YNIEGQZSON7XHHL7S3K6KNNJV4YR5FHQ3PA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1veqoBGw6a8AGMoigyuNJMjcEvCT93xRq2B1d2KNhs6aEmSqVF): protocol: expected piece hash; context canceled; EOF Dec 8 01:55:09.216085662 (58h26m9.087168707s) [noise=true , address=45.91.116.6:4003 , piece_id=CRONTV2B2AVZTLKU3YNIEGQZSON7XHHL7S3K6KNNJV4YR5FHQ3PA, seg_pos={0 0}, upload=51329]: Long tail cancellation due to optimal threshold Dec 8 01:55:09.216307705 (58h26m9.08739075s) [noise=true , address=145.79.227.30:4786 , piece_id=VM65RUN3GUZZFUOHFYE7ZLPWU6COCQUMY3IK7GCNWPIUZAMEBR7Q, seg_pos={0 0}, upload=51329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125TWU7R4pkmrUxsNLhgTD475aLbjUJ7psA6drJtE7oDqZXYJ4Q): protocol: expected piece hash; context canceled; EOF Dec 8 01:55:09.216313956 (58h26m9.087396991s) [noise=true , address=145.79.227.30:4786 , piece_id=VM65RUN3GUZZFUOHFYE7ZLPWU6COCQUMY3IK7GCNWPIUZAMEBR7Q, seg_pos={0 0}, upload=51329]: Long tail cancellation due to optimal threshold Dec 8 01:55:09.216414809 (58h26m9.087497844s) [noise=true , address=77.90.31.16:4760 , piece_id=Q67LYTDJCAR2TFVSAIIYCUYM7D44JFPBLCSKRNLXH26EAW5UQ3GA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1e6p1TD5CqUyWdZjkrxh2kRqMuyDiMUr9eimfeayDAKwhtBrZ8): protocol: expected piece hash; context canceled; EOF Dec 8 01:55:09.216420681 (58h26m9.087503726s) [noise=true , address=77.90.31.16:4760 , piece_id=Q67LYTDJCAR2TFVSAIIYCUYM7D44JFPBLCSKRNLXH26EAW5UQ3GA, seg_pos={0 0}, upload=51329]: Long tail cancellation due to optimal threshold Dec 8 01:55:09.216562861 (58h26m9.087645906s) [noise=true , address=45.91.116.7:4037 , piece_id=UBNFDSXQBISC5QQSNGAAZIR7LRE7FCYRVYN5MMOY3A2ALEOQBYQQ, seg_pos={0 0}, upload=51329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iNPwE2XSfXcaXAXcrH2jcyNGYeDiiWvzAcDQffrmjPwTVqVmK): protocol: expected piece hash; context canceled; EOF Dec 8 01:55:09.216570025 (58h26m9.08765307s) [noise=true , address=45.91.116.7:4037 , piece_id=UBNFDSXQBISC5QQSNGAAZIR7LRE7FCYRVYN5MMOY3A2ALEOQBYQQ, seg_pos={0 0}, upload=51329]: Long tail cancellation due to optimal threshold Dec 8 01:55:09.216585564 (58h26m9.087668609s) [seg_pos={0 0}, upload=51329]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 8 01:55:09.216654365 (58h26m9.0877374s) [upload=51329]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 01:55:09.216659064 (58h26m9.087742099s) [upload=51329]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 01:55:35.852213564 (58h26m35.723296629s) [upload=51329]: Done uploading segments... Dec 8 01:55:35.852273619 (58h26m35.723356664s) [upload=51329]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.751320181s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 4.86677979s trace: https://jaeger.dev.storj.io/trace/7af096edea7ffcf5 FAIL (2025-12-08 02:20:42 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.866810627s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 5.871812863s trace: https://jaeger.dev.storj.io/trace/27d92c196f473eec FAIL (2025-12-08 02:20:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 4891s) [upload=51469]: Got next segment. Inline: false Dec 8 02:04:21.143134547 (58h35m21.014217592s) [upload=51469]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 02:04:21.143141259 (58h35m21.014224304s) [upload=51469]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 02:04:22.267502706 (58h35m22.138585751s) [seg_pos={0 0}, upload=51469]: Begin upload segment... Dec 8 02:04:22.267624127 (58h35m22.138707172s) [seg_pos={0 0}, upload=51469]: Done begin upload segment. Dec 8 02:04:22.267698027 (58h35m22.138781073s) [noise=true , address=77.90.31.11:4554 , piece_id=BQP7XTRA2SBVINT4J7JAOF3ORNENBBIQN4ABMCEMQQ6FQL6QUOKA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.267712997 (58h35m22.138796042s) [noise=true , address=145.79.227.41:4087 , piece_id=XO6K3CXK7KZ5VEVXULHKQXLSSLIIWIX3VKQOSD7HY25WMNMCDLEQ, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.268434680 (58h35m22.139517715s) [noise=true , address=145.79.227.16:4910 , piece_id=ETE7ZBHT5IJUOBWCA5BGLKC2ANDEUB2M3DOMXWWS2RBWSJQBARRQ, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.268550331 (58h35m22.139633366s) [noise=true , address=145.79.227.26:4653 , piece_id=DKW72PEWPIOHPYHLI54PBU7NR3HGUBLFM3MN4XQQFWIIZ4FHIUNA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.268913111 (58h35m22.139996146s) [noise=true , address=145.79.227.23:4545 , piece_id=NBYCSU2KSHTAL223AHKHREQWKXUEQKJU46TY3SPYTKT5N5T2OUSA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.268924032 (58h35m22.140007067s) [noise=true , address=145.79.227.40:4051 , piece_id=7CQTGHFZ542WUHYX2CTOPN4PF2KDOOLDGV64THDBZCYOCXA3JWDA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.269398886 (58h35m22.140481921s) [noise=true , address=77.90.31.10:4518 , piece_id=6AR3NAFODP54SKG3SJSJSIQQFOYXO2C65WVNN4Z4XG426ZEOEDQA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.269414686 (58h35m22.140497721s) [noise=true , address=145.79.227.25:4619 , piece_id=XBNGVRIQLQUWKKBF4ULL7ZJDDT75QLY7ES2J7WHYWRD2ZLHOGU2Q, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.269753261 (58h35m22.140836296s) [noise=true , address=145.79.227.23:4547 , piece_id=5Y6V5QWLCWNOLT2GDNUEIRSG75KGNGIFLBCBGSIT7XH46RZYVDGQ, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.270016322 (58h35m22.141099367s) [noise=true , address=77.90.31.13:4628 , piece_id=LWTOCQRRA7HLI5ZPPRQ25IM22LMQNRR5H3TTOTSTGCFOYFZMABAA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.270259175 (58h35m22.14134222s) [noise=true , address=145.79.227.25:4617 , piece_id=S4ETCJ6WMCPXXQCYPPBBMHLMK2SVJV4S5VOKHGTDAGSGG6WRYH3Q, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.270655339 (58h35m22.141738374s) [noise=true , address=145.79.227.38:4977 , piece_id=73WXA33TWM4MFVTHDHW3PXJVABMCVMQYI3P42AHT3OYPKMXIVTUQ, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.271143158 (58h35m22.142246412s) [noise=true , address=145.79.227.28:4727 , piece_id=UYRK457JBF2AHCIXZ4XOIL7YHOVYZ7IPDUTB6WOZZUDEW5Q7NNCQ, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.271230565 (58h35m22.14231361s) [noise=true , address=145.79.227.39:4013 , piece_id=UI22RDXJ2DY3RFNSJVNT43XQJ6B2S3EO6SFLYHXW4VPVQJSYYF6Q, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.271662086 (58h35m22.142745121s) [noise=true , address=145.79.227.22:4511 , piece_id=GGA463DAOSZCW7SQGZW2LGFRQVNUOUJ2L2GXRXLSDPC65KVPY3TA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.272592658 (58h35m22.143675703s) [noise=true , address=145.79.227.24:4583 , piece_id=WYTXGXG77ZW4RGWX7YWDU2NH6R5CCPVBPQH3ISHUAJVKZJRONS5Q, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:24.009199123 (58h35m23.880282158s) [upload=51469]: Next returned nil segment Dec 8 02:04:24.009221235 (58h35m23.88030428s) [upload=51469]: Waiting for error group managing segments... Dec 8 02:04:24.293906896 (58h35m24.164989931s) [noise=true , address=145.79.227.24:4583 , piece_id=WYTXGXG77ZW4RGWX7YWDU2NH6R5CCPVBPQH3ISHUAJVKZJRONS5Q, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.298576738 (58h35m24.169659783s) [noise=true , address=145.79.227.28:4727 , piece_id=UYRK457JBF2AHCIXZ4XOIL7YHOVYZ7IPDUTB6WOZZUDEW5Q7NNCQ, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.301356571 (58h35m24.172439616s) [noise=true , address=145.79.227.25:4619 , piece_id=XBNGVRIQLQUWKKBF4ULL7ZJDDT75QLY7ES2J7WHYWRD2ZLHOGU2Q, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.326466765 (58h35m24.19754981s) [noise=true , address=145.79.227.16:4910 , piece_id=ETE7ZBHT5IJUOBWCA5BGLKC2ANDEUB2M3DOMXWWS2RBWSJQBARRQ, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.328633530 (58h35m24.199716586s) [noise=true , address=145.79.227.26:4653 , piece_id=DKW72PEWPIOHPYHLI54PBU7NR3HGUBLFM3MN4XQQFWIIZ4FHIUNA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.360508106 (58h35m24.231591151s) [noise=true , address=145.79.227.40:4051 , piece_id=7CQTGHFZ542WUHYX2CTOPN4PF2KDOOLDGV64THDBZCYOCXA3JWDA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.361116263 (58h35m24.232199308s) [noise=true , address=145.79.227.25:4617 , piece_id=S4ETCJ6WMCPXXQCYPPBBMHLMK2SVJV4S5VOKHGTDAGSGG6WRYH3Q, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.361695086 (58h35m24.232778131s) [noise=true , address=145.79.227.41:4087 , piece_id=XO6K3CXK7KZ5VEVXULHKQXLSSLIIWIX3VKQOSD7HY25WMNMCDLEQ, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.371520525 (58h35m24.24260356s) [noise=true , address=145.79.227.38:4977 , piece_id=73WXA33TWM4MFVTHDHW3PXJVABMCVMQYI3P42AHT3OYPKMXIVTUQ, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.381271893 (58h35m24.252354928s) [noise=true , address=145.79.227.23:4545 , piece_id=NBYCSU2KSHTAL223AHKHREQWKXUEQKJU46TY3SPYTKT5N5T2OUSA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.386403053 (58h35m24.257486098s) [noise=true , address=77.90.31.11:4554 , piece_id=BQP7XTRA2SBVINT4J7JAOF3ORNENBBIQN4ABMCEMQQ6FQL6QUOKA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.386427209 (58h35m24.257510254s) [noise=true , address=77.90.31.10:4518 , piece_id=6AR3NAFODP54SKG3SJSJSIQQFOYXO2C65WVNN4Z4XG426ZEOEDQA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.400489386 (58h35m24.271572431s) [noise=true , address=77.90.31.13:4628 , piece_id=LWTOCQRRA7HLI5ZPPRQ25IM22LMQNRR5H3TTOTSTGCFOYFZMABAA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.400513903 (58h35m24.271596938s) [seg_pos={0 0}, upload=51469]: Segment reached optimal threshold of 13 pieces. Dec 8 02:04:24.400813233 (58h35m24.271896288s) [noise=true , address=145.79.227.39:4013 , piece_id=UI22RDXJ2DY3RFNSJVNT43XQJ6B2S3EO6SFLYHXW4VPVQJSYYF6Q, seg_pos={0 0}, upload=51469]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b): protocol: write tcp 188.166.225.170:39434->145.79.227.39:4013: use of closed network connection; write tcp 188.166.225.170:39434->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39434->145.79.227.39:4013: use of closed network connection Dec 8 02:04:24.400825677 (58h35m24.271908722s) [noise=true , address=145.79.227.39:4013 , piece_id=UI22RDXJ2DY3RFNSJVNT43XQJ6B2S3EO6SFLYHXW4VPVQJSYYF6Q, seg_pos={0 0}, upload=51469]: Long tail cancellation due to optimal threshold Dec 8 02:04:24.400873027 (58h35m24.271956082s) [noise=true , address=145.79.227.22:4511 , piece_id=GGA463DAOSZCW7SQGZW2LGFRQVNUOUJ2L2GXRXLSDPC65KVPY3TA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG): protocol: write tcp 188.166.225.170:47926->145.79.227.22:4511: use of closed network connection; write tcp 188.166.225.170:47926->145.79.227.22:4511: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47926->145.79.227.22:4511: use of closed network connection Dec 8 02:04:24.400894248 (58h35m24.271977293s) [noise=true , address=145.79.227.22:4511 , piece_id=GGA463DAOSZCW7SQGZW2LGFRQVNUOUJ2L2GXRXLSDPC65KVPY3TA, seg_pos={0 0}, upload=51469]: Long tail cancellation due to optimal threshold Dec 8 02:04:24.401247129 (58h35m24.272330184s) [noise=true , address=145.79.227.23:4547 , piece_id=5Y6V5QWLCWNOLT2GDNUEIRSG75KGNGIFLBCBGSIT7XH46RZYVDGQ, seg_pos={0 0}, upload=51469]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW): protocol: read tcp 188.166.225.170:43344->145.79.227.23:4547: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 02:04:24.401260003 (58h35m24.272343059s) [noise=true , address=145.79.227.23:4547 , piece_id=5Y6V5QWLCWNOLT2GDNUEIRSG75KGNGIFLBCBGSIT7XH46RZYVDGQ, seg_pos={0 0}, upload=51469]: Long tail cancellation due to optimal threshold Dec 8 02:04:24.401285452 (58h35m24.272368497s) [seg_pos={0 0}, upload=51469]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 02:04:24.401382247 (58h35m24.272465292s) [upload=51469]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 02:04:24.401391394 (58h35m24.272474429s) [upload=51469]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 02:05:39.049922629 (58h36m38.921005704s) [upload=51469]: Done uploading segments... Dec 8 02:05:39.050003342 (58h36m38.921086377s) [upload=51469]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.871851427s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 4.933640858s trace: https://jaeger.dev.storj.io/trace/7af096edea7ffcf5 FAIL (2025-12-08 02:20:26 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.933671707s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [X..........................X......X................X....X...] duration: 3.457314865s trace: https://jaeger.dev.storj.io/trace/491130a80ad889c4 FAIL (2025-12-08 02:20:22 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.45734362s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........X................X.........X.......................] duration: 6.248325713s trace: https://jaeger.dev.storj.io/trace/e7164007ac0fd55 FAIL (2025-12-08 02:20:14 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 6.248354577s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 7.852276596s trace: https://jaeger.dev.storj.io/trace/57258282a8e974fd FAIL (2025-12-08 02:20:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== UA4TAJ2FSYO5S4NXECVMMMCXKFL5CAHPQBQJ6WREQ3YFC3Q, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.202906305 (58h26m7.07398934s) [noise=true , address=45.91.116.4:4945 , piece_id=6UNWQQAWWWWC2XVUCDRCTAYJDTHBXAZJMXA6KLCTOP5PYSAJCCGA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.203371201 (58h26m7.074454246s) [noise=true , address=145.79.227.11:4740 , piece_id=KL54LL2UQRT6MH2K3NQQJB3BKPX4QRWQMXRK27MW2U7CHFDCCWFA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.203861354 (58h26m7.07494439s) [noise=true , address=77.90.31.12:4621 , piece_id=S4FVJHEPIO2MVO56ZRIJ27RMTV723MERPLJLVD2T2GAO3T7K24TA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.204290401 (58h26m7.075373446s) [noise=true , address=145.79.227.34:4928 , piece_id=DRKF2DRSCBAYSZO66A2ZON2JS6HIJIT2YUWFZRS3W2CKFTV3NLVA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.204711363 (58h26m7.075794398s) [noise=true , address=145.79.227.15:4908 , piece_id=JIUZSMOAKBFCFAX6RQ52TCNZMYBREQCYOEM75U3H2QYR4WH3KWCA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.205118208 (58h26m7.076201253s) [noise=true , address=77.90.31.18:4825 , piece_id=EJ4LO4TCEFB4OQQLFOEUYGQKK462ZPXXEIIY5CHF2PERXWDGQNDA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.205554277 (58h26m7.076637323s) [noise=true , address=145.79.227.30:4786 , piece_id=VM65RUN3GUZZFUOHFYE7ZLPWU6COCQUMY3IK7GCNWPIUZAMEBR7Q, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.206116137 (58h26m7.077199182s) [noise=true , address=145.79.227.25:4627 , piece_id=K7UYU3W2CACUAQUUL72GG5F7DP5CHAHZ4M47TZG6NBN5Z6ZCDRBA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.206664562 (58h26m7.077747607s) [noise=true , address=77.90.31.16:4760 , piece_id=Q67LYTDJCAR2TFVSAIIYCUYM7D44JFPBLCSKRNLXH26EAW5UQ3GA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.207102556 (58h26m7.078185601s) [noise=true , address=45.91.116.7:4037 , piece_id=UBNFDSXQBISC5QQSNGAAZIR7LRE7FCYRVYN5MMOY3A2ALEOQBYQQ, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.207519911 (58h26m7.078602956s) [noise=true , address=77.90.31.11:4579 , piece_id=JINRNEVUAP2VI5Y4CR7ALWDQIAIAFX3SP2VVZSXDDG6XPUKASE3Q, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.207967393 (58h26m7.079050438s) [noise=true , address=77.90.31.10:4550 , piece_id=PDCZ7VXNHQJ6LYPG64GZRYP4MRBGVPXCWTXSRT3UV2N6HGMDE2KA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.208416427 (58h26m7.079499472s) [noise=true , address=45.91.116.6:4003 , piece_id=CRONTV2B2AVZTLKU3YNIEGQZSON7XHHL7S3K6KNNJV4YR5FHQ3PA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.208830746 (58h26m7.079913791s) [noise=true , address=45.91.116.12:4254 , piece_id=UCKU4TBXJC4PSYHRDPUXETXLAILDOWA5NN7RCVVNYLVUBY7FBIFA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.209255245 (58h26m7.080338291s) [noise=true , address=145.79.227.16:4933 , piece_id=LHF64L3SWXFVZ5X5D67I44WSJNKRFB5G4YP7DFHDWOAYE23II3OA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.209662921 (58h26m7.080745966s) [noise=true , address=145.79.227.18:4010 , piece_id=S2EOTEV4TBNNDOBN7TIX76AJE2WI5P5DM3JK2NJB25TISYLS7VSQ, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.214493680 (58h26m7.085576725s) [noise=true , address=77.90.31.10:4539 , piece_id=27YFK55QLZDEULYABQ3OBDSPDDEIUJ2MJSD7X3A4K6HRFJMRQJUA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:08.555121654 (58h26m8.426204699s) [upload=51329]: Next returned nil segment Dec 8 01:55:08.555145519 (58h26m8.426228564s) [upload=51329]: Waiting for error group managing segments... Dec 8 01:55:08.935193756 (58h26m8.806276791s) [noise=true , address=64.49.13.27:4957 , piece_id=7METNUA4TAJ2FSYO5S4NXECVMMMCXKFL5CAHPQBQJ6WREQ3YFC3Q, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.093051314 (58h26m8.964134359s) [noise=true , address=145.79.227.16:4933 , piece_id=LHF64L3SWXFVZ5X5D67I44WSJNKRFB5G4YP7DFHDWOAYE23II3OA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.098812273 (58h26m8.969895318s) [noise=true , address=145.79.227.18:4010 , piece_id=S2EOTEV4TBNNDOBN7TIX76AJE2WI5P5DM3JK2NJB25TISYLS7VSQ, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.102616718 (58h26m8.973699763s) [noise=true , address=145.79.227.25:4627 , piece_id=K7UYU3W2CACUAQUUL72GG5F7DP5CHAHZ4M47TZG6NBN5Z6ZCDRBA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.118902949 (58h26m8.989985994s) [noise=true , address=145.79.227.34:4928 , piece_id=DRKF2DRSCBAYSZO66A2ZON2JS6HIJIT2YUWFZRS3W2CKFTV3NLVA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.124783838 (58h26m8.995866883s) [noise=true , address=77.90.31.18:4825 , piece_id=EJ4LO4TCEFB4OQQLFOEUYGQKK462ZPXXEIIY5CHF2PERXWDGQNDA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.126718282 (58h26m8.997801327s) [noise=true , address=145.79.227.11:4740 , piece_id=KL54LL2UQRT6MH2K3NQQJB3BKPX4QRWQMXRK27MW2U7CHFDCCWFA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.145158485 (58h26m9.01624153s) [noise=true , address=77.90.31.10:4550 , piece_id=PDCZ7VXNHQJ6LYPG64GZRYP4MRBGVPXCWTXSRT3UV2N6HGMDE2KA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.156650047 (58h26m9.027733092s) [noise=true , address=145.79.227.15:4908 , piece_id=JIUZSMOAKBFCFAX6RQ52TCNZMYBREQCYOEM75U3H2QYR4WH3KWCA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.162020363 (58h26m9.033103398s) [noise=true , address=77.90.31.10:4539 , piece_id=27YFK55QLZDEULYABQ3OBDSPDDEIUJ2MJSD7X3A4K6HRFJMRQJUA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.198899907 (58h26m9.069982942s) [noise=true , address=77.90.31.12:4621 , piece_id=S4FVJHEPIO2MVO56ZRIJ27RMTV723MERPLJLVD2T2GAO3T7K24TA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.208468337 (58h26m9.079551382s) [noise=true , address=45.91.116.4:4945 , piece_id=6UNWQQAWWWWC2XVUCDRCTAYJDTHBXAZJMXA6KLCTOP5PYSAJCCGA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.212217697 (58h26m9.083300732s) [noise=true , address=45.91.116.12:4254 , piece_id=UCKU4TBXJC4PSYHRDPUXETXLAILDOWA5NN7RCVVNYLVUBY7FBIFA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.215832390 (58h26m9.086915435s) [noise=true , address=77.90.31.11:4579 , piece_id=JINRNEVUAP2VI5Y4CR7ALWDQIAIAFX3SP2VVZSXDDG6XPUKASE3Q, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.215862086 (58h26m9.086945161s) [seg_pos={0 0}, upload=51329]: Segment reached optimal threshold of 14 pieces. Dec 8 01:55:09.216078098 (58h26m9.087161133s) [noise=true , address=45.91.116.6:4003 , piece_id=CRONTV2B2AVZTLKU3YNIEGQZSON7XHHL7S3K6KNNJV4YR5FHQ3PA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1veqoBGw6a8AGMoigyuNJMjcEvCT93xRq2B1d2KNhs6aEmSqVF): protocol: expected piece hash; context canceled; EOF Dec 8 01:55:09.216085662 (58h26m9.087168707s) [noise=true , address=45.91.116.6:4003 , piece_id=CRONTV2B2AVZTLKU3YNIEGQZSON7XHHL7S3K6KNNJV4YR5FHQ3PA, seg_pos={0 0}, upload=51329]: Long tail cancellation due to optimal threshold Dec 8 01:55:09.216307705 (58h26m9.08739075s) [noise=true , address=145.79.227.30:4786 , piece_id=VM65RUN3GUZZFUOHFYE7ZLPWU6COCQUMY3IK7GCNWPIUZAMEBR7Q, seg_pos={0 0}, upload=51329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125TWU7R4pkmrUxsNLhgTD475aLbjUJ7psA6drJtE7oDqZXYJ4Q): protocol: expected piece hash; context canceled; EOF Dec 8 01:55:09.216313956 (58h26m9.087396991s) [noise=true , address=145.79.227.30:4786 , piece_id=VM65RUN3GUZZFUOHFYE7ZLPWU6COCQUMY3IK7GCNWPIUZAMEBR7Q, seg_pos={0 0}, upload=51329]: Long tail cancellation due to optimal threshold Dec 8 01:55:09.216414809 (58h26m9.087497844s) [noise=true , address=77.90.31.16:4760 , piece_id=Q67LYTDJCAR2TFVSAIIYCUYM7D44JFPBLCSKRNLXH26EAW5UQ3GA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1e6p1TD5CqUyWdZjkrxh2kRqMuyDiMUr9eimfeayDAKwhtBrZ8): protocol: expected piece hash; context canceled; EOF Dec 8 01:55:09.216420681 (58h26m9.087503726s) [noise=true , address=77.90.31.16:4760 , piece_id=Q67LYTDJCAR2TFVSAIIYCUYM7D44JFPBLCSKRNLXH26EAW5UQ3GA, seg_pos={0 0}, upload=51329]: Long tail cancellation due to optimal threshold Dec 8 01:55:09.216562861 (58h26m9.087645906s) [noise=true , address=45.91.116.7:4037 , piece_id=UBNFDSXQBISC5QQSNGAAZIR7LRE7FCYRVYN5MMOY3A2ALEOQBYQQ, seg_pos={0 0}, upload=51329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iNPwE2XSfXcaXAXcrH2jcyNGYeDiiWvzAcDQffrmjPwTVqVmK): protocol: expected piece hash; context canceled; EOF Dec 8 01:55:09.216570025 (58h26m9.08765307s) [noise=true , address=45.91.116.7:4037 , piece_id=UBNFDSXQBISC5QQSNGAAZIR7LRE7FCYRVYN5MMOY3A2ALEOQBYQQ, seg_pos={0 0}, upload=51329]: Long tail cancellation due to optimal threshold Dec 8 01:55:09.216585564 (58h26m9.087668609s) [seg_pos={0 0}, upload=51329]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 8 01:55:09.216654365 (58h26m9.0877374s) [upload=51329]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 01:55:09.216659064 (58h26m9.087742099s) [upload=51329]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 01:55:35.852213564 (58h26m35.723296629s) [upload=51329]: Done uploading segments... Dec 8 01:55:35.852273619 (58h26m35.723356664s) [upload=51329]: Upload finished. err: metaclient: context deadline exceeded Duration: 7.852312946s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 4.927020402s trace: https://jaeger.dev.storj.io/trace/7af096edea7ffcf5 FAIL (2025-12-08 02:19:56 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.92704549s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 6.644154575s trace: https://jaeger.dev.storj.io/trace/57258282a8e974fd FAIL (2025-12-08 02:19:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== UA4TAJ2FSYO5S4NXECVMMMCXKFL5CAHPQBQJ6WREQ3YFC3Q, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.202906305 (58h26m7.07398934s) [noise=true , address=45.91.116.4:4945 , piece_id=6UNWQQAWWWWC2XVUCDRCTAYJDTHBXAZJMXA6KLCTOP5PYSAJCCGA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.203371201 (58h26m7.074454246s) [noise=true , address=145.79.227.11:4740 , piece_id=KL54LL2UQRT6MH2K3NQQJB3BKPX4QRWQMXRK27MW2U7CHFDCCWFA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.203861354 (58h26m7.07494439s) [noise=true , address=77.90.31.12:4621 , piece_id=S4FVJHEPIO2MVO56ZRIJ27RMTV723MERPLJLVD2T2GAO3T7K24TA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.204290401 (58h26m7.075373446s) [noise=true , address=145.79.227.34:4928 , piece_id=DRKF2DRSCBAYSZO66A2ZON2JS6HIJIT2YUWFZRS3W2CKFTV3NLVA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.204711363 (58h26m7.075794398s) [noise=true , address=145.79.227.15:4908 , piece_id=JIUZSMOAKBFCFAX6RQ52TCNZMYBREQCYOEM75U3H2QYR4WH3KWCA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.205118208 (58h26m7.076201253s) [noise=true , address=77.90.31.18:4825 , piece_id=EJ4LO4TCEFB4OQQLFOEUYGQKK462ZPXXEIIY5CHF2PERXWDGQNDA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.205554277 (58h26m7.076637323s) [noise=true , address=145.79.227.30:4786 , piece_id=VM65RUN3GUZZFUOHFYE7ZLPWU6COCQUMY3IK7GCNWPIUZAMEBR7Q, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.206116137 (58h26m7.077199182s) [noise=true , address=145.79.227.25:4627 , piece_id=K7UYU3W2CACUAQUUL72GG5F7DP5CHAHZ4M47TZG6NBN5Z6ZCDRBA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.206664562 (58h26m7.077747607s) [noise=true , address=77.90.31.16:4760 , piece_id=Q67LYTDJCAR2TFVSAIIYCUYM7D44JFPBLCSKRNLXH26EAW5UQ3GA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.207102556 (58h26m7.078185601s) [noise=true , address=45.91.116.7:4037 , piece_id=UBNFDSXQBISC5QQSNGAAZIR7LRE7FCYRVYN5MMOY3A2ALEOQBYQQ, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.207519911 (58h26m7.078602956s) [noise=true , address=77.90.31.11:4579 , piece_id=JINRNEVUAP2VI5Y4CR7ALWDQIAIAFX3SP2VVZSXDDG6XPUKASE3Q, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.207967393 (58h26m7.079050438s) [noise=true , address=77.90.31.10:4550 , piece_id=PDCZ7VXNHQJ6LYPG64GZRYP4MRBGVPXCWTXSRT3UV2N6HGMDE2KA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.208416427 (58h26m7.079499472s) [noise=true , address=45.91.116.6:4003 , piece_id=CRONTV2B2AVZTLKU3YNIEGQZSON7XHHL7S3K6KNNJV4YR5FHQ3PA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.208830746 (58h26m7.079913791s) [noise=true , address=45.91.116.12:4254 , piece_id=UCKU4TBXJC4PSYHRDPUXETXLAILDOWA5NN7RCVVNYLVUBY7FBIFA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.209255245 (58h26m7.080338291s) [noise=true , address=145.79.227.16:4933 , piece_id=LHF64L3SWXFVZ5X5D67I44WSJNKRFB5G4YP7DFHDWOAYE23II3OA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.209662921 (58h26m7.080745966s) [noise=true , address=145.79.227.18:4010 , piece_id=S2EOTEV4TBNNDOBN7TIX76AJE2WI5P5DM3JK2NJB25TISYLS7VSQ, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.214493680 (58h26m7.085576725s) [noise=true , address=77.90.31.10:4539 , piece_id=27YFK55QLZDEULYABQ3OBDSPDDEIUJ2MJSD7X3A4K6HRFJMRQJUA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:08.555121654 (58h26m8.426204699s) [upload=51329]: Next returned nil segment Dec 8 01:55:08.555145519 (58h26m8.426228564s) [upload=51329]: Waiting for error group managing segments... Dec 8 01:55:08.935193756 (58h26m8.806276791s) [noise=true , address=64.49.13.27:4957 , piece_id=7METNUA4TAJ2FSYO5S4NXECVMMMCXKFL5CAHPQBQJ6WREQ3YFC3Q, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.093051314 (58h26m8.964134359s) [noise=true , address=145.79.227.16:4933 , piece_id=LHF64L3SWXFVZ5X5D67I44WSJNKRFB5G4YP7DFHDWOAYE23II3OA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.098812273 (58h26m8.969895318s) [noise=true , address=145.79.227.18:4010 , piece_id=S2EOTEV4TBNNDOBN7TIX76AJE2WI5P5DM3JK2NJB25TISYLS7VSQ, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.102616718 (58h26m8.973699763s) [noise=true , address=145.79.227.25:4627 , piece_id=K7UYU3W2CACUAQUUL72GG5F7DP5CHAHZ4M47TZG6NBN5Z6ZCDRBA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.118902949 (58h26m8.989985994s) [noise=true , address=145.79.227.34:4928 , piece_id=DRKF2DRSCBAYSZO66A2ZON2JS6HIJIT2YUWFZRS3W2CKFTV3NLVA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.124783838 (58h26m8.995866883s) [noise=true , address=77.90.31.18:4825 , piece_id=EJ4LO4TCEFB4OQQLFOEUYGQKK462ZPXXEIIY5CHF2PERXWDGQNDA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.126718282 (58h26m8.997801327s) [noise=true , address=145.79.227.11:4740 , piece_id=KL54LL2UQRT6MH2K3NQQJB3BKPX4QRWQMXRK27MW2U7CHFDCCWFA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.145158485 (58h26m9.01624153s) [noise=true , address=77.90.31.10:4550 , piece_id=PDCZ7VXNHQJ6LYPG64GZRYP4MRBGVPXCWTXSRT3UV2N6HGMDE2KA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.156650047 (58h26m9.027733092s) [noise=true , address=145.79.227.15:4908 , piece_id=JIUZSMOAKBFCFAX6RQ52TCNZMYBREQCYOEM75U3H2QYR4WH3KWCA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.162020363 (58h26m9.033103398s) [noise=true , address=77.90.31.10:4539 , piece_id=27YFK55QLZDEULYABQ3OBDSPDDEIUJ2MJSD7X3A4K6HRFJMRQJUA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.198899907 (58h26m9.069982942s) [noise=true , address=77.90.31.12:4621 , piece_id=S4FVJHEPIO2MVO56ZRIJ27RMTV723MERPLJLVD2T2GAO3T7K24TA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.208468337 (58h26m9.079551382s) [noise=true , address=45.91.116.4:4945 , piece_id=6UNWQQAWWWWC2XVUCDRCTAYJDTHBXAZJMXA6KLCTOP5PYSAJCCGA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.212217697 (58h26m9.083300732s) [noise=true , address=45.91.116.12:4254 , piece_id=UCKU4TBXJC4PSYHRDPUXETXLAILDOWA5NN7RCVVNYLVUBY7FBIFA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.215832390 (58h26m9.086915435s) [noise=true , address=77.90.31.11:4579 , piece_id=JINRNEVUAP2VI5Y4CR7ALWDQIAIAFX3SP2VVZSXDDG6XPUKASE3Q, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.215862086 (58h26m9.086945161s) [seg_pos={0 0}, upload=51329]: Segment reached optimal threshold of 14 pieces. Dec 8 01:55:09.216078098 (58h26m9.087161133s) [noise=true , address=45.91.116.6:4003 , piece_id=CRONTV2B2AVZTLKU3YNIEGQZSON7XHHL7S3K6KNNJV4YR5FHQ3PA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1veqoBGw6a8AGMoigyuNJMjcEvCT93xRq2B1d2KNhs6aEmSqVF): protocol: expected piece hash; context canceled; EOF Dec 8 01:55:09.216085662 (58h26m9.087168707s) [noise=true , address=45.91.116.6:4003 , piece_id=CRONTV2B2AVZTLKU3YNIEGQZSON7XHHL7S3K6KNNJV4YR5FHQ3PA, seg_pos={0 0}, upload=51329]: Long tail cancellation due to optimal threshold Dec 8 01:55:09.216307705 (58h26m9.08739075s) [noise=true , address=145.79.227.30:4786 , piece_id=VM65RUN3GUZZFUOHFYE7ZLPWU6COCQUMY3IK7GCNWPIUZAMEBR7Q, seg_pos={0 0}, upload=51329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125TWU7R4pkmrUxsNLhgTD475aLbjUJ7psA6drJtE7oDqZXYJ4Q): protocol: expected piece hash; context canceled; EOF Dec 8 01:55:09.216313956 (58h26m9.087396991s) [noise=true , address=145.79.227.30:4786 , piece_id=VM65RUN3GUZZFUOHFYE7ZLPWU6COCQUMY3IK7GCNWPIUZAMEBR7Q, seg_pos={0 0}, upload=51329]: Long tail cancellation due to optimal threshold Dec 8 01:55:09.216414809 (58h26m9.087497844s) [noise=true , address=77.90.31.16:4760 , piece_id=Q67LYTDJCAR2TFVSAIIYCUYM7D44JFPBLCSKRNLXH26EAW5UQ3GA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1e6p1TD5CqUyWdZjkrxh2kRqMuyDiMUr9eimfeayDAKwhtBrZ8): protocol: expected piece hash; context canceled; EOF Dec 8 01:55:09.216420681 (58h26m9.087503726s) [noise=true , address=77.90.31.16:4760 , piece_id=Q67LYTDJCAR2TFVSAIIYCUYM7D44JFPBLCSKRNLXH26EAW5UQ3GA, seg_pos={0 0}, upload=51329]: Long tail cancellation due to optimal threshold Dec 8 01:55:09.216562861 (58h26m9.087645906s) [noise=true , address=45.91.116.7:4037 , piece_id=UBNFDSXQBISC5QQSNGAAZIR7LRE7FCYRVYN5MMOY3A2ALEOQBYQQ, seg_pos={0 0}, upload=51329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iNPwE2XSfXcaXAXcrH2jcyNGYeDiiWvzAcDQffrmjPwTVqVmK): protocol: expected piece hash; context canceled; EOF Dec 8 01:55:09.216570025 (58h26m9.08765307s) [noise=true , address=45.91.116.7:4037 , piece_id=UBNFDSXQBISC5QQSNGAAZIR7LRE7FCYRVYN5MMOY3A2ALEOQBYQQ, seg_pos={0 0}, upload=51329]: Long tail cancellation due to optimal threshold Dec 8 01:55:09.216585564 (58h26m9.087668609s) [seg_pos={0 0}, upload=51329]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 8 01:55:09.216654365 (58h26m9.0877374s) [upload=51329]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 01:55:09.216659064 (58h26m9.087742099s) [upload=51329]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 01:55:35.852213564 (58h26m35.723296629s) [upload=51329]: Done uploading segments... Dec 8 01:55:35.852273619 (58h26m35.723356664s) [upload=51329]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.644185915s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)