Orbiter: EU/US/AP/EDGE | Code | Config | Playbook | Escalation Procedure
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [..................................X.........................] duration:
27.559001158s trace:
https://jaeger.dev.storj.io/trace/6820217d73ecf2a
FAIL (2025-12-26 08:53:59 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
ss=45.91.116.13:4174 , piece_id=RLKVQ3QJM4STACAOUTWA7STEIG4PITJRIB4YOI3GNTYMPA76IFHQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.705106031 (497h3m36.576189076s) [noise=true , address=77.90.31.19:4870 , piece_id=ZHJNUA7P36RBWXQXMDLFRIHQPSZNOPGL67AQVDTDEPCRY2R4EMWQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.705493759 (497h3m36.576576804s) [noise=true , address=145.79.227.20:4443 , piece_id=HAFVT75GMRHFRR5E5VD4AOJFHGRUS56MQX3HNHI6FQYYMZT4J4FQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.705630279 (497h3m36.576713334s) [noise=true , address=77.90.31.15:4717 , piece_id=CVL5BLBIJ5DWSL7ZSEJ4I62RENOFPNWWNZCSE62CTMWGSS2XTE2Q, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.705923348 (497h3m36.577006393s) [noise=true , address=77.90.31.12:4607 , piece_id=TPFAMNN2BEYPSYRYOMDMSS77ZRB35HOJYWMJGTJEERRWGV5KQHHA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706055580 (497h3m36.577138615s) [noise=true , address=64.49.13.169:4705 , piece_id=NG53QDRTA5IAV5W64W6MSZVNCWDUDHBB3DP4BF6G7DTF5DAF7ECA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706362764 (497h3m36.577445819s) [noise=true , address=77.90.31.11:4581 , piece_id=NQOGQYJLH7W7CXFLBDW3XUN6Y6YVQEWAFBMLNKNHKTU5VCV4V3GQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706507651 (497h3m36.577590686s) [noise=true , address=145.79.227.44:4226 , piece_id=XFE2F2254TY5CMO7J5O6S3RDKHOHSLQCUGDNHB37VKOXPM63GJEA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706728992 (497h3m36.577812027s) [noise=true , address=45.91.116.4:4942 , piece_id=VDKHG53ATFYJDQYDBXKZ7SN2RNAAZBYGQVGAF34PCG4UYO7XMFGA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706869399 (497h3m36.577952444s) [noise=true , address=77.90.31.16:4750 , piece_id=HIZWD4OHAZ6UKMODPFHABQRQW7ESEZPVMSD45RI5HGXPLHDJLJHQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707077395 (497h3m36.57816044s) [noise=true , address=145.79.227.34:4909 , piece_id=TSBSOK42K3BBLLTUBV6UFWW33FK67FZHJZTKEQFAAQ7ITKY3RAEQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707229524 (497h3m36.578312559s) [noise=true , address=77.90.31.13:4655 , piece_id=B6QS3X24QQRPWA3O5ENXR32XJOJZBZZOVVDMSESJYKU3XBIPGDIQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707443251 (497h3m36.578526296s) [noise=true , address=145.79.227.5:4241 , piece_id=ANDNWFXV5YWOH55WDLOMJXSKZVN45XSIYT2I4QB52DNO2PT3HYKQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707633815 (497h3m36.57871685s) [noise=true , address=64.49.13.154:4930 , piece_id=IHQYIUPLXMAKL4S76Y6NAGDP2ZWFRQBOXXJJENZ6JRGBXWLKDCFQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707813797 (497h3m36.578896832s) [noise=true , address=145.79.227.13:4822 , piece_id=HSHVLD6IYQOG3KQYCZGENM4ZYH7XCKGRAZLEL6N2WELIBI2LNHSA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.708022374 (497h3m36.579105439s) [noise=true , address=145.79.227.30:4775 , piece_id=OG65SPMQI3XHNJHUHKSJ4NZKGAB764OYFE7D5CBJCP3YRAGYANQA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.708682712 (497h3m36.579765747s) [noise=true , address=77.90.31.17:4788 , piece_id=CIK5XZI3QMG7LFEOKBPZGNLDOORNP6IQV3QYSHLIY5XSF3LLXBBQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:38.013251430 (497h3m37.884334475s) [upload=436256]: Next returned nil segment
Dec 26 08:32:38.013277480 (497h3m37.884360525s) [upload=436256]: Waiting for error group managing segments...
Dec 26 08:32:38.345076384 (497h3m38.216159419s) [noise=true , address=64.49.13.154:4930 , piece_id=IHQYIUPLXMAKL4S76Y6NAGDP2ZWFRQBOXXJJENZ6JRGBXWLKDCFQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.593234781 (497h3m38.464317836s) [noise=true , address=145.79.227.20:4443 , piece_id=HAFVT75GMRHFRR5E5VD4AOJFHGRUS56MQX3HNHI6FQYYMZT4J4FQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.593901300 (497h3m38.464984335s) [noise=true , address=145.79.227.5:4241 , piece_id=ANDNWFXV5YWOH55WDLOMJXSKZVN45XSIYT2I4QB52DNO2PT3HYKQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.594508065 (497h3m38.4655911s) [noise=true , address=145.79.227.13:4822 , piece_id=HSHVLD6IYQOG3KQYCZGENM4ZYH7XCKGRAZLEL6N2WELIBI2LNHSA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.636070587 (497h3m38.507153653s) [noise=true , address=145.79.227.34:4909 , piece_id=TSBSOK42K3BBLLTUBV6UFWW33FK67FZHJZTKEQFAAQ7ITKY3RAEQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.750629848 (497h3m38.621712883s) [noise=true , address=77.90.31.11:4581 , piece_id=NQOGQYJLH7W7CXFLBDW3XUN6Y6YVQEWAFBMLNKNHKTU5VCV4V3GQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.814987737 (497h3m38.686070782s) [noise=true , address=45.91.116.4:4942 , piece_id=VDKHG53ATFYJDQYDBXKZ7SN2RNAAZBYGQVGAF34PCG4UYO7XMFGA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.847327759 (497h3m38.718410794s) [noise=true , address=77.90.31.15:4717 , piece_id=CVL5BLBIJ5DWSL7ZSEJ4I62RENOFPNWWNZCSE62CTMWGSS2XTE2Q, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.902089014 (497h3m38.773172049s) [noise=true , address=45.91.116.13:4174 , piece_id=RLKVQ3QJM4STACAOUTWA7STEIG4PITJRIB4YOI3GNTYMPA76IFHQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.912841608 (497h3m38.783924644s) [noise=true , address=77.90.31.12:4607 , piece_id=TPFAMNN2BEYPSYRYOMDMSS77ZRB35HOJYWMJGTJEERRWGV5KQHHA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.070851326 (497h3m38.941934371s) [noise=true , address=77.90.31.16:4750 , piece_id=HIZWD4OHAZ6UKMODPFHABQRQW7ESEZPVMSD45RI5HGXPLHDJLJHQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.469018898 (497h3m39.340101933s) [noise=true , address=145.79.227.30:4775 , piece_id=OG65SPMQI3XHNJHUHKSJ4NZKGAB764OYFE7D5CBJCP3YRAGYANQA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.477948602 (497h3m39.349031637s) [noise=true , address=145.79.227.44:4226 , piece_id=XFE2F2254TY5CMO7J5O6S3RDKHOHSLQCUGDNHB37VKOXPM63GJEA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.534878005 (497h3m39.40596105s) [noise=true , address=77.90.31.17:4788 , piece_id=CIK5XZI3QMG7LFEOKBPZGNLDOORNP6IQV3QYSHLIY5XSF3LLXBBQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.534902853 (497h3m39.405985898s) [seg_pos={0 0}, upload=436256]: Segment reached optimal threshold of 14 pieces.
Dec 26 08:32:39.535131698 (497h3m39.406214743s) [noise=true , address=77.90.31.19:4870 , piece_id=ZHJNUA7P36RBWXQXMDLFRIHQPSZNOPGL67AQVDTDEPCRY2R4EMWQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hNVXEVe8DW8F8meERGWkxAazEZKLKBeprNn6e5C94eEoAMBaR): protocol: expected piece hash; context canceled; EOF
Dec 26 08:32:39.535140075 (497h3m39.40622311s) [noise=true , address=77.90.31.19:4870 , piece_id=ZHJNUA7P36RBWXQXMDLFRIHQPSZNOPGL67AQVDTDEPCRY2R4EMWQ, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold
Dec 26 08:32:39.535367978 (497h3m39.406451013s) [noise=true , address=77.90.31.18:4838 , piece_id=V5LDUMURFU4H43E2A5MTLWBI6ZJHVPYMUILBDV5N7ERELPLTIBAA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1if7vf1cNTBwAhb6owi6tuGLYqSfa3feWhsvGSevwuwykhz2RY): protocol: expected piece hash; context canceled; EOF
Dec 26 08:32:39.535375542 (497h3m39.406458577s) [noise=true , address=77.90.31.18:4838 , piece_id=V5LDUMURFU4H43E2A5MTLWBI6ZJHVPYMUILBDV5N7ERELPLTIBAA, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold
Dec 26 08:32:39.535565575 (497h3m39.40664861s) [noise=true , address=77.90.31.13:4655 , piece_id=B6QS3X24QQRPWA3O5ENXR32XJOJZBZZOVVDMSESJYKU3XBIPGDIQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1K7fQM1NVA1L7ukmDtyWh2DYWypUoj3GbreETZpPL6gXfXmZDD): protocol: expected piece hash; context canceled; EOF
Dec 26 08:32:39.535606041 (497h3m39.406689077s) [noise=true , address=77.90.31.13:4655 , piece_id=B6QS3X24QQRPWA3O5ENXR32XJOJZBZZOVVDMSESJYKU3XBIPGDIQ, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold
Dec 26 08:32:39.535707996 (497h3m39.406791031s) [noise=true , address=64.49.13.169:4705 , piece_id=NG53QDRTA5IAV5W64W6MSZVNCWDUDHBB3DP4BF6G7DTF5DAF7ECA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: read tcp 188.166.225.170:35726->64.49.13.169:4705: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 26 08:32:39.535715059 (497h3m39.406798094s) [noise=true , address=64.49.13.169:4705 , piece_id=NG53QDRTA5IAV5W64W6MSZVNCWDUDHBB3DP4BF6G7DTF5DAF7ECA, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold
Dec 26 08:32:39.535740878 (497h3m39.406823923s) [seg_pos={0 0}, upload=436256]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 26 08:32:39.535861157 (497h3m39.406944202s) [upload=436256]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 08:32:39.535867620 (497h3m39.406950655s) [upload=436256]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 08:32:39.833497059 (497h3m39.704580164s) [upload=436256]: Done uploading segments...
Dec 26 08:32:39.833522858 (497h3m39.704605903s) [upload=436256]: Upload finished. err: <nil>
Duration: 27.559016157s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [X...........................................................] duration:
8.78498774s trace:
https://jaeger.dev.storj.io/trace/4b8b74dd4906ed4
FAIL (2025-12-26 08:53:59 UTC)
orbiter warning: context deadline exceeded
Duration: 8.785007067s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
19 out of 60 is failed (threshold=3) [XX.X.X..X..X.XXX.....X......X.X...X.XX..X.XXX...............] duration:
5.766599927s trace:
https://jaeger.dev.storj.io/trace/3312cc3668ef2c5f
FAIL (2025-12-26 08:53:56 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 26 08:42:18.506798665 (497h13m18.37788171s) [upload=436400]: Starting upload
Dec 26 08:42:18.506881102 (497h13m18.377964137s) [upload=436400]: Done starting upload
Dec 26 08:42:18.506963669 (497h13m18.378046714s) [upload=436400]: Uploading segments...
Dec 26 08:42:18.507170413 (497h13m18.378253478s) [upload=436400]: Got next segment. Inline: false
Dec 26 08:42:18.507247920 (497h13m18.378330965s) [upload=436400]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 26 08:42:18.507255454 (497h13m18.378338499s) [upload=436400]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 26 08:42:18.754283459 (497h13m18.625366504s) [seg_pos={0 0}, upload=436400]: Begin upload segment...
Dec 26 08:42:18.754356328 (497h13m18.625439373s) [seg_pos={0 0}, upload=436400]: Done begin upload segment.
Dec 26 08:42:18.754391745 (497h13m18.62547479s) [noise=true , address=145.79.227.40:4051 , piece_id=NLPGNO63SJLYA65HYN2CJBJFS6OTLX7ISULHSM6Q54PWOOTVAVWQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.754466177 (497h13m18.625549222s) [noise=true , address=77.90.31.17:4773 , piece_id=P645R4U4WSCWJR5XZSGFGVVF3QHAVSP6SN46CPQGBT6JWJB57ZIQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.755567554 (497h13m18.626650599s) [noise=true , address=77.90.31.18:4808 , piece_id=INW543LSVQLUQQ4FZZXVTVPV2LSGPKMUVHCSKE3OQ2CQAIEQFD2Q, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.755910858 (497h13m18.626993893s) [noise=true , address=145.79.227.17:4970 , piece_id=JXWYFTTKYNB6BSPBKWBK4YPF2PNAGWJKXHJAOUFIJVVIKIVNPUJA, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.756229915 (497h13m18.62731296s) [noise=true , address=145.79.227.46:4603 , piece_id=7L43455SH4JVSXBJYL3WVMRCWFQZ2CEI6DQA7MUXDYSKPPKESY7A, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.756613886 (497h13m18.627696922s) [noise=true , address=145.79.227.16:4912 , piece_id=TIPMXPQDX4HVH2NUQ3FUFT6J34SNM5DT2NHDXCZO76MMGPEEEHMQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.757020902 (497h13m18.628103937s) [noise=true , address=145.79.227.38:4977 , piece_id=MYMT3XBOXWU5DZOTPVHSVVXOKNHW7WWVCBECIO37VAYUTFK7XZSQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.757428849 (497h13m18.628511904s) [noise=true , address=77.90.31.11:4555 , piece_id=ZTSV63NMIHDFU464BAUD3MH3645MJOCGLTDQEKIRPD47EMLLO5EA, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.757902380 (497h13m18.628985425s) [noise=true , address=145.79.227.45:4229 , piece_id=GNXPJKQLV6ITICMBLE4PY2ODOKEJ6WWBUAPA74AO4IBVOVVEFSLA, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.757979276 (497h13m18.629062321s) [noise=true , address=77.90.31.21:4878 , piece_id=RNZ3EK2ZERZY6IQEBAZPHEUDBHNF7BARJF5FVZMLVVMLH5OK7WCQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.758365802 (497h13m18.629448837s) [noise=true , address=45.91.116.4:4937 , piece_id=7HT3WA63RZNVO3AX775OL37N4BJRXZCZ5P6A2A2Z6NPZX6RASSSQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.758773228 (497h13m18.629856263s) [noise=true , address=77.90.31.10:4517 , piece_id=RD3ARVUTAOQRS3YUQ7WL2LF47CXQFSVDDJSJUAEZAFVGUU4HJ7KQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.759342462 (497h13m18.630425507s) [noise=true , address=145.79.227.37:4567 , piece_id=J7Z3543ZQ2TIGFFHBO7V7GJKTGOY54Z6WAPYDI7ALYRYDS6NEK2Q, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.760026635 (497h13m18.63110968s) [noise=true , address=145.79.227.48:4301 , piece_id=IS2W2WAK7NOMKFEHY3QK7KS35XPQYC2723Q2ZJ27HTX4FPBQYHDQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.760798124 (497h13m18.631881169s) [noise=true , address=145.79.227.51:4411 , piece_id=7JDMY4F74SKL6NSFN2HO36EGUCM5MWVWR5HLXACHBLZXTDRRWI3Q, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.761677319 (497h13m18.632760364s) [noise=true , address=77.90.31.14:4664 , piece_id=JFEBDNYO2OQRCKCLBKWS43SZCZ7L2LSIF5FP2JS37FFARUEHCESA, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:20.466851608 (497h13m20.337934643s) [upload=436400]: Next returned nil segment
Dec 26 08:42:20.466865273 (497h13m20.337948318s) [upload=436400]: Waiting for error group managing segments...
Dec 26 08:42:20.750402218 (497h13m20.621485253s) [noise=true , address=145.79.227.37:4567 , piece_id=J7Z3543ZQ2TIGFFHBO7V7GJKTGOY54Z6WAPYDI7ALYRYDS6NEK2Q, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:20.763579740 (497h13m20.634662785s) [noise=true , address=145.79.227.48:4301 , piece_id=IS2W2WAK7NOMKFEHY3QK7KS35XPQYC2723Q2ZJ27HTX4FPBQYHDQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:20.767844120 (497h13m20.638927155s) [noise=true , address=145.79.227.38:4977 , piece_id=MYMT3XBOXWU5DZOTPVHSVVXOKNHW7WWVCBECIO37VAYUTFK7XZSQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:20.771941653 (497h13m20.643024698s) [noise=true , address=145.79.227.16:4912 , piece_id=TIPMXPQDX4HVH2NUQ3FUFT6J34SNM5DT2NHDXCZO76MMGPEEEHMQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:20.773887688 (497h13m20.644970723s) [noise=true , address=145.79.227.17:4970 , piece_id=JXWYFTTKYNB6BSPBKWBK4YPF2PNAGWJKXHJAOUFIJVVIKIVNPUJA, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:20.781414641 (497h13m20.652497676s) [noise=true , address=145.79.227.51:4411 , piece_id=7JDMY4F74SKL6NSFN2HO36EGUCM5MWVWR5HLXACHBLZXTDRRWI3Q, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:20.781591407 (497h13m20.652674452s) [noise=true , address=145.79.227.46:4603 , piece_id=7L43455SH4JVSXBJYL3WVMRCWFQZ2CEI6DQA7MUXDYSKPPKESY7A, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:20.787415879 (497h13m20.658498924s) [noise=true , address=145.79.227.45:4229 , piece_id=GNXPJKQLV6ITICMBLE4PY2ODOKEJ6WWBUAPA74AO4IBVOVVEFSLA, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:21.029225186 (497h13m20.900308221s) [noise=true , address=45.91.116.4:4937 , piece_id=7HT3WA63RZNVO3AX775OL37N4BJRXZCZ5P6A2A2Z6NPZX6RASSSQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:21.124342636 (497h13m20.995425671s) [noise=true , address=77.90.31.21:4878 , piece_id=RNZ3EK2ZERZY6IQEBAZPHEUDBHNF7BARJF5FVZMLVVMLH5OK7WCQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:21.148939092 (497h13m21.020022127s) [noise=true , address=77.90.31.14:4664 , piece_id=JFEBDNYO2OQRCKCLBKWS43SZCZ7L2LSIF5FP2JS37FFARUEHCESA, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:21.733228030 (497h13m21.604311075s) [noise=true , address=77.90.31.11:4555 , piece_id=ZTSV63NMIHDFU464BAUD3MH3645MJOCGLTDQEKIRPD47EMLLO5EA, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:21.818455067 (497h13m21.689538102s) [noise=true , address=77.90.31.17:4773 , piece_id=P645R4U4WSCWJR5XZSGFGVVF3QHAVSP6SN46CPQGBT6JWJB57ZIQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:21.818474194 (497h13m21.689557239s) [seg_pos={0 0}, upload=436400]: Segment reached optimal threshold of 13 pieces.
Dec 26 08:42:21.818682150 (497h13m21.689765195s) [noise=true , address=77.90.31.10:4517 , piece_id=RD3ARVUTAOQRS3YUQ7WL2LF47CXQFSVDDJSJUAEZAFVGUU4HJ7KQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF
Dec 26 08:42:21.818690926 (497h13m21.689773971s) [noise=true , address=77.90.31.10:4517 , piece_id=RD3ARVUTAOQRS3YUQ7WL2LF47CXQFSVDDJSJUAEZAFVGUU4HJ7KQ, seg_pos={0 0}, upload=436400]: Long tail cancellation due to optimal threshold
Dec 26 08:42:21.818911596 (497h13m21.689994631s) [noise=true , address=145.79.227.40:4051 , piece_id=NLPGNO63SJLYA65HYN2CJBJFS6OTLX7ISULHSM6Q54PWOOTVAVWQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF
Dec 26 08:42:21.818918088 (497h13m21.690001123s) [noise=true , address=145.79.227.40:4051 , piece_id=NLPGNO63SJLYA65HYN2CJBJFS6OTLX7ISULHSM6Q54PWOOTVAVWQ, seg_pos={0 0}, upload=436400]: Long tail cancellation due to optimal threshold
Dec 26 08:42:21.818941363 (497h13m21.690024418s) [noise=true , address=77.90.31.18:4808 , piece_id=INW543LSVQLUQQ4FZZXVTVPV2LSGPKMUVHCSKE3OQ2CQAIEQFD2Q, seg_pos={0 0}, upload=436400]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: expected piece hash; context canceled; EOF
Dec 26 08:42:21.818997390 (497h13m21.690080425s) [noise=true , address=77.90.31.18:4808 , piece_id=INW543LSVQLUQQ4FZZXVTVPV2LSGPKMUVHCSKE3OQ2CQAIEQFD2Q, seg_pos={0 0}, upload=436400]: Long tail cancellation due to optimal threshold
Dec 26 08:42:21.819032707 (497h13m21.690115742s) [seg_pos={0 0}, upload=436400]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 26 08:42:21.819224622 (497h13m21.690307657s) [upload=436400]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 08:42:21.819232357 (497h13m21.690315392s) [upload=436400]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 08:42:22.109308312 (497h13m21.980391357s) [upload=436400]: Done uploading segments...
Dec 26 08:42:22.109331417 (497h13m21.980414452s) [upload=436400]: Upload finished. err: <nil>
Duration: 5.76662302s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [...................................X........................] duration:
36.304196713s trace:
https://jaeger.dev.storj.io/trace/6820217d73ecf2a
FAIL (2025-12-26 08:53:31 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
ss=45.91.116.13:4174 , piece_id=RLKVQ3QJM4STACAOUTWA7STEIG4PITJRIB4YOI3GNTYMPA76IFHQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.705106031 (497h3m36.576189076s) [noise=true , address=77.90.31.19:4870 , piece_id=ZHJNUA7P36RBWXQXMDLFRIHQPSZNOPGL67AQVDTDEPCRY2R4EMWQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.705493759 (497h3m36.576576804s) [noise=true , address=145.79.227.20:4443 , piece_id=HAFVT75GMRHFRR5E5VD4AOJFHGRUS56MQX3HNHI6FQYYMZT4J4FQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.705630279 (497h3m36.576713334s) [noise=true , address=77.90.31.15:4717 , piece_id=CVL5BLBIJ5DWSL7ZSEJ4I62RENOFPNWWNZCSE62CTMWGSS2XTE2Q, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.705923348 (497h3m36.577006393s) [noise=true , address=77.90.31.12:4607 , piece_id=TPFAMNN2BEYPSYRYOMDMSS77ZRB35HOJYWMJGTJEERRWGV5KQHHA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706055580 (497h3m36.577138615s) [noise=true , address=64.49.13.169:4705 , piece_id=NG53QDRTA5IAV5W64W6MSZVNCWDUDHBB3DP4BF6G7DTF5DAF7ECA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706362764 (497h3m36.577445819s) [noise=true , address=77.90.31.11:4581 , piece_id=NQOGQYJLH7W7CXFLBDW3XUN6Y6YVQEWAFBMLNKNHKTU5VCV4V3GQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706507651 (497h3m36.577590686s) [noise=true , address=145.79.227.44:4226 , piece_id=XFE2F2254TY5CMO7J5O6S3RDKHOHSLQCUGDNHB37VKOXPM63GJEA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706728992 (497h3m36.577812027s) [noise=true , address=45.91.116.4:4942 , piece_id=VDKHG53ATFYJDQYDBXKZ7SN2RNAAZBYGQVGAF34PCG4UYO7XMFGA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706869399 (497h3m36.577952444s) [noise=true , address=77.90.31.16:4750 , piece_id=HIZWD4OHAZ6UKMODPFHABQRQW7ESEZPVMSD45RI5HGXPLHDJLJHQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707077395 (497h3m36.57816044s) [noise=true , address=145.79.227.34:4909 , piece_id=TSBSOK42K3BBLLTUBV6UFWW33FK67FZHJZTKEQFAAQ7ITKY3RAEQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707229524 (497h3m36.578312559s) [noise=true , address=77.90.31.13:4655 , piece_id=B6QS3X24QQRPWA3O5ENXR32XJOJZBZZOVVDMSESJYKU3XBIPGDIQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707443251 (497h3m36.578526296s) [noise=true , address=145.79.227.5:4241 , piece_id=ANDNWFXV5YWOH55WDLOMJXSKZVN45XSIYT2I4QB52DNO2PT3HYKQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707633815 (497h3m36.57871685s) [noise=true , address=64.49.13.154:4930 , piece_id=IHQYIUPLXMAKL4S76Y6NAGDP2ZWFRQBOXXJJENZ6JRGBXWLKDCFQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707813797 (497h3m36.578896832s) [noise=true , address=145.79.227.13:4822 , piece_id=HSHVLD6IYQOG3KQYCZGENM4ZYH7XCKGRAZLEL6N2WELIBI2LNHSA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.708022374 (497h3m36.579105439s) [noise=true , address=145.79.227.30:4775 , piece_id=OG65SPMQI3XHNJHUHKSJ4NZKGAB764OYFE7D5CBJCP3YRAGYANQA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.708682712 (497h3m36.579765747s) [noise=true , address=77.90.31.17:4788 , piece_id=CIK5XZI3QMG7LFEOKBPZGNLDOORNP6IQV3QYSHLIY5XSF3LLXBBQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:38.013251430 (497h3m37.884334475s) [upload=436256]: Next returned nil segment
Dec 26 08:32:38.013277480 (497h3m37.884360525s) [upload=436256]: Waiting for error group managing segments...
Dec 26 08:32:38.345076384 (497h3m38.216159419s) [noise=true , address=64.49.13.154:4930 , piece_id=IHQYIUPLXMAKL4S76Y6NAGDP2ZWFRQBOXXJJENZ6JRGBXWLKDCFQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.593234781 (497h3m38.464317836s) [noise=true , address=145.79.227.20:4443 , piece_id=HAFVT75GMRHFRR5E5VD4AOJFHGRUS56MQX3HNHI6FQYYMZT4J4FQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.593901300 (497h3m38.464984335s) [noise=true , address=145.79.227.5:4241 , piece_id=ANDNWFXV5YWOH55WDLOMJXSKZVN45XSIYT2I4QB52DNO2PT3HYKQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.594508065 (497h3m38.4655911s) [noise=true , address=145.79.227.13:4822 , piece_id=HSHVLD6IYQOG3KQYCZGENM4ZYH7XCKGRAZLEL6N2WELIBI2LNHSA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.636070587 (497h3m38.507153653s) [noise=true , address=145.79.227.34:4909 , piece_id=TSBSOK42K3BBLLTUBV6UFWW33FK67FZHJZTKEQFAAQ7ITKY3RAEQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.750629848 (497h3m38.621712883s) [noise=true , address=77.90.31.11:4581 , piece_id=NQOGQYJLH7W7CXFLBDW3XUN6Y6YVQEWAFBMLNKNHKTU5VCV4V3GQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.814987737 (497h3m38.686070782s) [noise=true , address=45.91.116.4:4942 , piece_id=VDKHG53ATFYJDQYDBXKZ7SN2RNAAZBYGQVGAF34PCG4UYO7XMFGA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.847327759 (497h3m38.718410794s) [noise=true , address=77.90.31.15:4717 , piece_id=CVL5BLBIJ5DWSL7ZSEJ4I62RENOFPNWWNZCSE62CTMWGSS2XTE2Q, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.902089014 (497h3m38.773172049s) [noise=true , address=45.91.116.13:4174 , piece_id=RLKVQ3QJM4STACAOUTWA7STEIG4PITJRIB4YOI3GNTYMPA76IFHQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.912841608 (497h3m38.783924644s) [noise=true , address=77.90.31.12:4607 , piece_id=TPFAMNN2BEYPSYRYOMDMSS77ZRB35HOJYWMJGTJEERRWGV5KQHHA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.070851326 (497h3m38.941934371s) [noise=true , address=77.90.31.16:4750 , piece_id=HIZWD4OHAZ6UKMODPFHABQRQW7ESEZPVMSD45RI5HGXPLHDJLJHQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.469018898 (497h3m39.340101933s) [noise=true , address=145.79.227.30:4775 , piece_id=OG65SPMQI3XHNJHUHKSJ4NZKGAB764OYFE7D5CBJCP3YRAGYANQA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.477948602 (497h3m39.349031637s) [noise=true , address=145.79.227.44:4226 , piece_id=XFE2F2254TY5CMO7J5O6S3RDKHOHSLQCUGDNHB37VKOXPM63GJEA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.534878005 (497h3m39.40596105s) [noise=true , address=77.90.31.17:4788 , piece_id=CIK5XZI3QMG7LFEOKBPZGNLDOORNP6IQV3QYSHLIY5XSF3LLXBBQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.534902853 (497h3m39.405985898s) [seg_pos={0 0}, upload=436256]: Segment reached optimal threshold of 14 pieces.
Dec 26 08:32:39.535131698 (497h3m39.406214743s) [noise=true , address=77.90.31.19:4870 , piece_id=ZHJNUA7P36RBWXQXMDLFRIHQPSZNOPGL67AQVDTDEPCRY2R4EMWQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hNVXEVe8DW8F8meERGWkxAazEZKLKBeprNn6e5C94eEoAMBaR): protocol: expected piece hash; context canceled; EOF
Dec 26 08:32:39.535140075 (497h3m39.40622311s) [noise=true , address=77.90.31.19:4870 , piece_id=ZHJNUA7P36RBWXQXMDLFRIHQPSZNOPGL67AQVDTDEPCRY2R4EMWQ, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold
Dec 26 08:32:39.535367978 (497h3m39.406451013s) [noise=true , address=77.90.31.18:4838 , piece_id=V5LDUMURFU4H43E2A5MTLWBI6ZJHVPYMUILBDV5N7ERELPLTIBAA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1if7vf1cNTBwAhb6owi6tuGLYqSfa3feWhsvGSevwuwykhz2RY): protocol: expected piece hash; context canceled; EOF
Dec 26 08:32:39.535375542 (497h3m39.406458577s) [noise=true , address=77.90.31.18:4838 , piece_id=V5LDUMURFU4H43E2A5MTLWBI6ZJHVPYMUILBDV5N7ERELPLTIBAA, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold
Dec 26 08:32:39.535565575 (497h3m39.40664861s) [noise=true , address=77.90.31.13:4655 , piece_id=B6QS3X24QQRPWA3O5ENXR32XJOJZBZZOVVDMSESJYKU3XBIPGDIQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1K7fQM1NVA1L7ukmDtyWh2DYWypUoj3GbreETZpPL6gXfXmZDD): protocol: expected piece hash; context canceled; EOF
Dec 26 08:32:39.535606041 (497h3m39.406689077s) [noise=true , address=77.90.31.13:4655 , piece_id=B6QS3X24QQRPWA3O5ENXR32XJOJZBZZOVVDMSESJYKU3XBIPGDIQ, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold
Dec 26 08:32:39.535707996 (497h3m39.406791031s) [noise=true , address=64.49.13.169:4705 , piece_id=NG53QDRTA5IAV5W64W6MSZVNCWDUDHBB3DP4BF6G7DTF5DAF7ECA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: read tcp 188.166.225.170:35726->64.49.13.169:4705: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 26 08:32:39.535715059 (497h3m39.406798094s) [noise=true , address=64.49.13.169:4705 , piece_id=NG53QDRTA5IAV5W64W6MSZVNCWDUDHBB3DP4BF6G7DTF5DAF7ECA, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold
Dec 26 08:32:39.535740878 (497h3m39.406823923s) [seg_pos={0 0}, upload=436256]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 26 08:32:39.535861157 (497h3m39.406944202s) [upload=436256]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 08:32:39.535867620 (497h3m39.406950655s) [upload=436256]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 08:32:39.833497059 (497h3m39.704580164s) [upload=436256]: Done uploading segments...
Dec 26 08:32:39.833522858 (497h3m39.704605903s) [upload=436256]: Upload finished. err: <nil>
Duration: 36.304222573s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
19 out of 60 is failed (threshold=3) [.XX.X.X..X..X.XXX.....X......X.X...X.XX..X.XXX..............] duration:
6.339672107s trace:
https://jaeger.dev.storj.io/trace/3312cc3668ef2c5f
FAIL (2025-12-26 08:53:15 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 26 08:42:18.506798665 (497h13m18.37788171s) [upload=436400]: Starting upload
Dec 26 08:42:18.506881102 (497h13m18.377964137s) [upload=436400]: Done starting upload
Dec 26 08:42:18.506963669 (497h13m18.378046714s) [upload=436400]: Uploading segments...
Dec 26 08:42:18.507170413 (497h13m18.378253478s) [upload=436400]: Got next segment. Inline: false
Dec 26 08:42:18.507247920 (497h13m18.378330965s) [upload=436400]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 26 08:42:18.507255454 (497h13m18.378338499s) [upload=436400]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 26 08:42:18.754283459 (497h13m18.625366504s) [seg_pos={0 0}, upload=436400]: Begin upload segment...
Dec 26 08:42:18.754356328 (497h13m18.625439373s) [seg_pos={0 0}, upload=436400]: Done begin upload segment.
Dec 26 08:42:18.754391745 (497h13m18.62547479s) [noise=true , address=145.79.227.40:4051 , piece_id=NLPGNO63SJLYA65HYN2CJBJFS6OTLX7ISULHSM6Q54PWOOTVAVWQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.754466177 (497h13m18.625549222s) [noise=true , address=77.90.31.17:4773 , piece_id=P645R4U4WSCWJR5XZSGFGVVF3QHAVSP6SN46CPQGBT6JWJB57ZIQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.755567554 (497h13m18.626650599s) [noise=true , address=77.90.31.18:4808 , piece_id=INW543LSVQLUQQ4FZZXVTVPV2LSGPKMUVHCSKE3OQ2CQAIEQFD2Q, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.755910858 (497h13m18.626993893s) [noise=true , address=145.79.227.17:4970 , piece_id=JXWYFTTKYNB6BSPBKWBK4YPF2PNAGWJKXHJAOUFIJVVIKIVNPUJA, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.756229915 (497h13m18.62731296s) [noise=true , address=145.79.227.46:4603 , piece_id=7L43455SH4JVSXBJYL3WVMRCWFQZ2CEI6DQA7MUXDYSKPPKESY7A, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.756613886 (497h13m18.627696922s) [noise=true , address=145.79.227.16:4912 , piece_id=TIPMXPQDX4HVH2NUQ3FUFT6J34SNM5DT2NHDXCZO76MMGPEEEHMQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.757020902 (497h13m18.628103937s) [noise=true , address=145.79.227.38:4977 , piece_id=MYMT3XBOXWU5DZOTPVHSVVXOKNHW7WWVCBECIO37VAYUTFK7XZSQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.757428849 (497h13m18.628511904s) [noise=true , address=77.90.31.11:4555 , piece_id=ZTSV63NMIHDFU464BAUD3MH3645MJOCGLTDQEKIRPD47EMLLO5EA, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.757902380 (497h13m18.628985425s) [noise=true , address=145.79.227.45:4229 , piece_id=GNXPJKQLV6ITICMBLE4PY2ODOKEJ6WWBUAPA74AO4IBVOVVEFSLA, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.757979276 (497h13m18.629062321s) [noise=true , address=77.90.31.21:4878 , piece_id=RNZ3EK2ZERZY6IQEBAZPHEUDBHNF7BARJF5FVZMLVVMLH5OK7WCQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.758365802 (497h13m18.629448837s) [noise=true , address=45.91.116.4:4937 , piece_id=7HT3WA63RZNVO3AX775OL37N4BJRXZCZ5P6A2A2Z6NPZX6RASSSQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.758773228 (497h13m18.629856263s) [noise=true , address=77.90.31.10:4517 , piece_id=RD3ARVUTAOQRS3YUQ7WL2LF47CXQFSVDDJSJUAEZAFVGUU4HJ7KQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.759342462 (497h13m18.630425507s) [noise=true , address=145.79.227.37:4567 , piece_id=J7Z3543ZQ2TIGFFHBO7V7GJKTGOY54Z6WAPYDI7ALYRYDS6NEK2Q, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.760026635 (497h13m18.63110968s) [noise=true , address=145.79.227.48:4301 , piece_id=IS2W2WAK7NOMKFEHY3QK7KS35XPQYC2723Q2ZJ27HTX4FPBQYHDQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.760798124 (497h13m18.631881169s) [noise=true , address=145.79.227.51:4411 , piece_id=7JDMY4F74SKL6NSFN2HO36EGUCM5MWVWR5HLXACHBLZXTDRRWI3Q, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.761677319 (497h13m18.632760364s) [noise=true , address=77.90.31.14:4664 , piece_id=JFEBDNYO2OQRCKCLBKWS43SZCZ7L2LSIF5FP2JS37FFARUEHCESA, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:20.466851608 (497h13m20.337934643s) [upload=436400]: Next returned nil segment
Dec 26 08:42:20.466865273 (497h13m20.337948318s) [upload=436400]: Waiting for error group managing segments...
Dec 26 08:42:20.750402218 (497h13m20.621485253s) [noise=true , address=145.79.227.37:4567 , piece_id=J7Z3543ZQ2TIGFFHBO7V7GJKTGOY54Z6WAPYDI7ALYRYDS6NEK2Q, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:20.763579740 (497h13m20.634662785s) [noise=true , address=145.79.227.48:4301 , piece_id=IS2W2WAK7NOMKFEHY3QK7KS35XPQYC2723Q2ZJ27HTX4FPBQYHDQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:20.767844120 (497h13m20.638927155s) [noise=true , address=145.79.227.38:4977 , piece_id=MYMT3XBOXWU5DZOTPVHSVVXOKNHW7WWVCBECIO37VAYUTFK7XZSQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:20.771941653 (497h13m20.643024698s) [noise=true , address=145.79.227.16:4912 , piece_id=TIPMXPQDX4HVH2NUQ3FUFT6J34SNM5DT2NHDXCZO76MMGPEEEHMQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:20.773887688 (497h13m20.644970723s) [noise=true , address=145.79.227.17:4970 , piece_id=JXWYFTTKYNB6BSPBKWBK4YPF2PNAGWJKXHJAOUFIJVVIKIVNPUJA, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:20.781414641 (497h13m20.652497676s) [noise=true , address=145.79.227.51:4411 , piece_id=7JDMY4F74SKL6NSFN2HO36EGUCM5MWVWR5HLXACHBLZXTDRRWI3Q, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:20.781591407 (497h13m20.652674452s) [noise=true , address=145.79.227.46:4603 , piece_id=7L43455SH4JVSXBJYL3WVMRCWFQZ2CEI6DQA7MUXDYSKPPKESY7A, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:20.787415879 (497h13m20.658498924s) [noise=true , address=145.79.227.45:4229 , piece_id=GNXPJKQLV6ITICMBLE4PY2ODOKEJ6WWBUAPA74AO4IBVOVVEFSLA, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:21.029225186 (497h13m20.900308221s) [noise=true , address=45.91.116.4:4937 , piece_id=7HT3WA63RZNVO3AX775OL37N4BJRXZCZ5P6A2A2Z6NPZX6RASSSQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:21.124342636 (497h13m20.995425671s) [noise=true , address=77.90.31.21:4878 , piece_id=RNZ3EK2ZERZY6IQEBAZPHEUDBHNF7BARJF5FVZMLVVMLH5OK7WCQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:21.148939092 (497h13m21.020022127s) [noise=true , address=77.90.31.14:4664 , piece_id=JFEBDNYO2OQRCKCLBKWS43SZCZ7L2LSIF5FP2JS37FFARUEHCESA, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:21.733228030 (497h13m21.604311075s) [noise=true , address=77.90.31.11:4555 , piece_id=ZTSV63NMIHDFU464BAUD3MH3645MJOCGLTDQEKIRPD47EMLLO5EA, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:21.818455067 (497h13m21.689538102s) [noise=true , address=77.90.31.17:4773 , piece_id=P645R4U4WSCWJR5XZSGFGVVF3QHAVSP6SN46CPQGBT6JWJB57ZIQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:21.818474194 (497h13m21.689557239s) [seg_pos={0 0}, upload=436400]: Segment reached optimal threshold of 13 pieces.
Dec 26 08:42:21.818682150 (497h13m21.689765195s) [noise=true , address=77.90.31.10:4517 , piece_id=RD3ARVUTAOQRS3YUQ7WL2LF47CXQFSVDDJSJUAEZAFVGUU4HJ7KQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF
Dec 26 08:42:21.818690926 (497h13m21.689773971s) [noise=true , address=77.90.31.10:4517 , piece_id=RD3ARVUTAOQRS3YUQ7WL2LF47CXQFSVDDJSJUAEZAFVGUU4HJ7KQ, seg_pos={0 0}, upload=436400]: Long tail cancellation due to optimal threshold
Dec 26 08:42:21.818911596 (497h13m21.689994631s) [noise=true , address=145.79.227.40:4051 , piece_id=NLPGNO63SJLYA65HYN2CJBJFS6OTLX7ISULHSM6Q54PWOOTVAVWQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF
Dec 26 08:42:21.818918088 (497h13m21.690001123s) [noise=true , address=145.79.227.40:4051 , piece_id=NLPGNO63SJLYA65HYN2CJBJFS6OTLX7ISULHSM6Q54PWOOTVAVWQ, seg_pos={0 0}, upload=436400]: Long tail cancellation due to optimal threshold
Dec 26 08:42:21.818941363 (497h13m21.690024418s) [noise=true , address=77.90.31.18:4808 , piece_id=INW543LSVQLUQQ4FZZXVTVPV2LSGPKMUVHCSKE3OQ2CQAIEQFD2Q, seg_pos={0 0}, upload=436400]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: expected piece hash; context canceled; EOF
Dec 26 08:42:21.818997390 (497h13m21.690080425s) [noise=true , address=77.90.31.18:4808 , piece_id=INW543LSVQLUQQ4FZZXVTVPV2LSGPKMUVHCSKE3OQ2CQAIEQFD2Q, seg_pos={0 0}, upload=436400]: Long tail cancellation due to optimal threshold
Dec 26 08:42:21.819032707 (497h13m21.690115742s) [seg_pos={0 0}, upload=436400]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 26 08:42:21.819224622 (497h13m21.690307657s) [upload=436400]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 08:42:21.819232357 (497h13m21.690315392s) [upload=436400]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 08:42:22.109308312 (497h13m21.980391357s) [upload=436400]: Done uploading segments...
Dec 26 08:42:22.109331417 (497h13m21.980414452s) [upload=436400]: Upload finished. err: <nil>
Duration: 6.339698757s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [X...........................................................] duration:
3.517985959s trace:
https://jaeger.dev.storj.io/trace/185fddc4dc40eba9
FAIL (2025-12-26 08:53:07 UTC)
orbiter warning: context deadline exceeded
Duration: 3.518014845s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
20 out of 60 is failed (threshold=3) [X.XX.X.X..X..X.XXX.....X......X.X...X.XX..X.XXX.............] duration:
5.9174189s trace:
https://jaeger.dev.storj.io/trace/3312cc3668ef2c5f
FAIL (2025-12-26 08:52:52 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 26 08:42:18.506798665 (497h13m18.37788171s) [upload=436400]: Starting upload
Dec 26 08:42:18.506881102 (497h13m18.377964137s) [upload=436400]: Done starting upload
Dec 26 08:42:18.506963669 (497h13m18.378046714s) [upload=436400]: Uploading segments...
Dec 26 08:42:18.507170413 (497h13m18.378253478s) [upload=436400]: Got next segment. Inline: false
Dec 26 08:42:18.507247920 (497h13m18.378330965s) [upload=436400]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 26 08:42:18.507255454 (497h13m18.378338499s) [upload=436400]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 26 08:42:18.754283459 (497h13m18.625366504s) [seg_pos={0 0}, upload=436400]: Begin upload segment...
Dec 26 08:42:18.754356328 (497h13m18.625439373s) [seg_pos={0 0}, upload=436400]: Done begin upload segment.
Dec 26 08:42:18.754391745 (497h13m18.62547479s) [noise=true , address=145.79.227.40:4051 , piece_id=NLPGNO63SJLYA65HYN2CJBJFS6OTLX7ISULHSM6Q54PWOOTVAVWQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.754466177 (497h13m18.625549222s) [noise=true , address=77.90.31.17:4773 , piece_id=P645R4U4WSCWJR5XZSGFGVVF3QHAVSP6SN46CPQGBT6JWJB57ZIQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.755567554 (497h13m18.626650599s) [noise=true , address=77.90.31.18:4808 , piece_id=INW543LSVQLUQQ4FZZXVTVPV2LSGPKMUVHCSKE3OQ2CQAIEQFD2Q, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.755910858 (497h13m18.626993893s) [noise=true , address=145.79.227.17:4970 , piece_id=JXWYFTTKYNB6BSPBKWBK4YPF2PNAGWJKXHJAOUFIJVVIKIVNPUJA, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.756229915 (497h13m18.62731296s) [noise=true , address=145.79.227.46:4603 , piece_id=7L43455SH4JVSXBJYL3WVMRCWFQZ2CEI6DQA7MUXDYSKPPKESY7A, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.756613886 (497h13m18.627696922s) [noise=true , address=145.79.227.16:4912 , piece_id=TIPMXPQDX4HVH2NUQ3FUFT6J34SNM5DT2NHDXCZO76MMGPEEEHMQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.757020902 (497h13m18.628103937s) [noise=true , address=145.79.227.38:4977 , piece_id=MYMT3XBOXWU5DZOTPVHSVVXOKNHW7WWVCBECIO37VAYUTFK7XZSQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.757428849 (497h13m18.628511904s) [noise=true , address=77.90.31.11:4555 , piece_id=ZTSV63NMIHDFU464BAUD3MH3645MJOCGLTDQEKIRPD47EMLLO5EA, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.757902380 (497h13m18.628985425s) [noise=true , address=145.79.227.45:4229 , piece_id=GNXPJKQLV6ITICMBLE4PY2ODOKEJ6WWBUAPA74AO4IBVOVVEFSLA, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.757979276 (497h13m18.629062321s) [noise=true , address=77.90.31.21:4878 , piece_id=RNZ3EK2ZERZY6IQEBAZPHEUDBHNF7BARJF5FVZMLVVMLH5OK7WCQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.758365802 (497h13m18.629448837s) [noise=true , address=45.91.116.4:4937 , piece_id=7HT3WA63RZNVO3AX775OL37N4BJRXZCZ5P6A2A2Z6NPZX6RASSSQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.758773228 (497h13m18.629856263s) [noise=true , address=77.90.31.10:4517 , piece_id=RD3ARVUTAOQRS3YUQ7WL2LF47CXQFSVDDJSJUAEZAFVGUU4HJ7KQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.759342462 (497h13m18.630425507s) [noise=true , address=145.79.227.37:4567 , piece_id=J7Z3543ZQ2TIGFFHBO7V7GJKTGOY54Z6WAPYDI7ALYRYDS6NEK2Q, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.760026635 (497h13m18.63110968s) [noise=true , address=145.79.227.48:4301 , piece_id=IS2W2WAK7NOMKFEHY3QK7KS35XPQYC2723Q2ZJ27HTX4FPBQYHDQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.760798124 (497h13m18.631881169s) [noise=true , address=145.79.227.51:4411 , piece_id=7JDMY4F74SKL6NSFN2HO36EGUCM5MWVWR5HLXACHBLZXTDRRWI3Q, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.761677319 (497h13m18.632760364s) [noise=true , address=77.90.31.14:4664 , piece_id=JFEBDNYO2OQRCKCLBKWS43SZCZ7L2LSIF5FP2JS37FFARUEHCESA, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:20.466851608 (497h13m20.337934643s) [upload=436400]: Next returned nil segment
Dec 26 08:42:20.466865273 (497h13m20.337948318s) [upload=436400]: Waiting for error group managing segments...
Dec 26 08:42:20.750402218 (497h13m20.621485253s) [noise=true , address=145.79.227.37:4567 , piece_id=J7Z3543ZQ2TIGFFHBO7V7GJKTGOY54Z6WAPYDI7ALYRYDS6NEK2Q, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:20.763579740 (497h13m20.634662785s) [noise=true , address=145.79.227.48:4301 , piece_id=IS2W2WAK7NOMKFEHY3QK7KS35XPQYC2723Q2ZJ27HTX4FPBQYHDQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:20.767844120 (497h13m20.638927155s) [noise=true , address=145.79.227.38:4977 , piece_id=MYMT3XBOXWU5DZOTPVHSVVXOKNHW7WWVCBECIO37VAYUTFK7XZSQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:20.771941653 (497h13m20.643024698s) [noise=true , address=145.79.227.16:4912 , piece_id=TIPMXPQDX4HVH2NUQ3FUFT6J34SNM5DT2NHDXCZO76MMGPEEEHMQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:20.773887688 (497h13m20.644970723s) [noise=true , address=145.79.227.17:4970 , piece_id=JXWYFTTKYNB6BSPBKWBK4YPF2PNAGWJKXHJAOUFIJVVIKIVNPUJA, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:20.781414641 (497h13m20.652497676s) [noise=true , address=145.79.227.51:4411 , piece_id=7JDMY4F74SKL6NSFN2HO36EGUCM5MWVWR5HLXACHBLZXTDRRWI3Q, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:20.781591407 (497h13m20.652674452s) [noise=true , address=145.79.227.46:4603 , piece_id=7L43455SH4JVSXBJYL3WVMRCWFQZ2CEI6DQA7MUXDYSKPPKESY7A, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:20.787415879 (497h13m20.658498924s) [noise=true , address=145.79.227.45:4229 , piece_id=GNXPJKQLV6ITICMBLE4PY2ODOKEJ6WWBUAPA74AO4IBVOVVEFSLA, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:21.029225186 (497h13m20.900308221s) [noise=true , address=45.91.116.4:4937 , piece_id=7HT3WA63RZNVO3AX775OL37N4BJRXZCZ5P6A2A2Z6NPZX6RASSSQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:21.124342636 (497h13m20.995425671s) [noise=true , address=77.90.31.21:4878 , piece_id=RNZ3EK2ZERZY6IQEBAZPHEUDBHNF7BARJF5FVZMLVVMLH5OK7WCQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:21.148939092 (497h13m21.020022127s) [noise=true , address=77.90.31.14:4664 , piece_id=JFEBDNYO2OQRCKCLBKWS43SZCZ7L2LSIF5FP2JS37FFARUEHCESA, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:21.733228030 (497h13m21.604311075s) [noise=true , address=77.90.31.11:4555 , piece_id=ZTSV63NMIHDFU464BAUD3MH3645MJOCGLTDQEKIRPD47EMLLO5EA, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:21.818455067 (497h13m21.689538102s) [noise=true , address=77.90.31.17:4773 , piece_id=P645R4U4WSCWJR5XZSGFGVVF3QHAVSP6SN46CPQGBT6JWJB57ZIQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:21.818474194 (497h13m21.689557239s) [seg_pos={0 0}, upload=436400]: Segment reached optimal threshold of 13 pieces.
Dec 26 08:42:21.818682150 (497h13m21.689765195s) [noise=true , address=77.90.31.10:4517 , piece_id=RD3ARVUTAOQRS3YUQ7WL2LF47CXQFSVDDJSJUAEZAFVGUU4HJ7KQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF
Dec 26 08:42:21.818690926 (497h13m21.689773971s) [noise=true , address=77.90.31.10:4517 , piece_id=RD3ARVUTAOQRS3YUQ7WL2LF47CXQFSVDDJSJUAEZAFVGUU4HJ7KQ, seg_pos={0 0}, upload=436400]: Long tail cancellation due to optimal threshold
Dec 26 08:42:21.818911596 (497h13m21.689994631s) [noise=true , address=145.79.227.40:4051 , piece_id=NLPGNO63SJLYA65HYN2CJBJFS6OTLX7ISULHSM6Q54PWOOTVAVWQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF
Dec 26 08:42:21.818918088 (497h13m21.690001123s) [noise=true , address=145.79.227.40:4051 , piece_id=NLPGNO63SJLYA65HYN2CJBJFS6OTLX7ISULHSM6Q54PWOOTVAVWQ, seg_pos={0 0}, upload=436400]: Long tail cancellation due to optimal threshold
Dec 26 08:42:21.818941363 (497h13m21.690024418s) [noise=true , address=77.90.31.18:4808 , piece_id=INW543LSVQLUQQ4FZZXVTVPV2LSGPKMUVHCSKE3OQ2CQAIEQFD2Q, seg_pos={0 0}, upload=436400]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: expected piece hash; context canceled; EOF
Dec 26 08:42:21.818997390 (497h13m21.690080425s) [noise=true , address=77.90.31.18:4808 , piece_id=INW543LSVQLUQQ4FZZXVTVPV2LSGPKMUVHCSKE3OQ2CQAIEQFD2Q, seg_pos={0 0}, upload=436400]: Long tail cancellation due to optimal threshold
Dec 26 08:42:21.819032707 (497h13m21.690115742s) [seg_pos={0 0}, upload=436400]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 26 08:42:21.819224622 (497h13m21.690307657s) [upload=436400]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 08:42:21.819232357 (497h13m21.690315392s) [upload=436400]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 08:42:22.109308312 (497h13m21.980391357s) [upload=436400]: Done uploading segments...
Dec 26 08:42:22.109331417 (497h13m21.980414452s) [upload=436400]: Upload finished. err: <nil>
Duration: 5.917449297s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.X..........................................................] duration:
5.784696203s trace:
https://jaeger.dev.storj.io/trace/4b8b74dd4906ed4
FAIL (2025-12-26 08:52:51 UTC)
orbiter warning: context deadline exceeded
Duration: 5.784729745s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
20 out of 60 is failed (threshold=3) [.X.XX.X.X..X..X.XXX.....X......X.X...X.XX..X.XXX............] duration:
5.981655175s trace:
https://jaeger.dev.storj.io/trace/3312cc3668ef2c5f
FAIL (2025-12-26 08:52:22 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 26 08:42:18.506798665 (497h13m18.37788171s) [upload=436400]: Starting upload
Dec 26 08:42:18.506881102 (497h13m18.377964137s) [upload=436400]: Done starting upload
Dec 26 08:42:18.506963669 (497h13m18.378046714s) [upload=436400]: Uploading segments...
Dec 26 08:42:18.507170413 (497h13m18.378253478s) [upload=436400]: Got next segment. Inline: false
Dec 26 08:42:18.507247920 (497h13m18.378330965s) [upload=436400]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 26 08:42:18.507255454 (497h13m18.378338499s) [upload=436400]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 26 08:42:18.754283459 (497h13m18.625366504s) [seg_pos={0 0}, upload=436400]: Begin upload segment...
Dec 26 08:42:18.754356328 (497h13m18.625439373s) [seg_pos={0 0}, upload=436400]: Done begin upload segment.
Dec 26 08:42:18.754391745 (497h13m18.62547479s) [noise=true , address=145.79.227.40:4051 , piece_id=NLPGNO63SJLYA65HYN2CJBJFS6OTLX7ISULHSM6Q54PWOOTVAVWQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.754466177 (497h13m18.625549222s) [noise=true , address=77.90.31.17:4773 , piece_id=P645R4U4WSCWJR5XZSGFGVVF3QHAVSP6SN46CPQGBT6JWJB57ZIQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.755567554 (497h13m18.626650599s) [noise=true , address=77.90.31.18:4808 , piece_id=INW543LSVQLUQQ4FZZXVTVPV2LSGPKMUVHCSKE3OQ2CQAIEQFD2Q, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.755910858 (497h13m18.626993893s) [noise=true , address=145.79.227.17:4970 , piece_id=JXWYFTTKYNB6BSPBKWBK4YPF2PNAGWJKXHJAOUFIJVVIKIVNPUJA, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.756229915 (497h13m18.62731296s) [noise=true , address=145.79.227.46:4603 , piece_id=7L43455SH4JVSXBJYL3WVMRCWFQZ2CEI6DQA7MUXDYSKPPKESY7A, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.756613886 (497h13m18.627696922s) [noise=true , address=145.79.227.16:4912 , piece_id=TIPMXPQDX4HVH2NUQ3FUFT6J34SNM5DT2NHDXCZO76MMGPEEEHMQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.757020902 (497h13m18.628103937s) [noise=true , address=145.79.227.38:4977 , piece_id=MYMT3XBOXWU5DZOTPVHSVVXOKNHW7WWVCBECIO37VAYUTFK7XZSQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.757428849 (497h13m18.628511904s) [noise=true , address=77.90.31.11:4555 , piece_id=ZTSV63NMIHDFU464BAUD3MH3645MJOCGLTDQEKIRPD47EMLLO5EA, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.757902380 (497h13m18.628985425s) [noise=true , address=145.79.227.45:4229 , piece_id=GNXPJKQLV6ITICMBLE4PY2ODOKEJ6WWBUAPA74AO4IBVOVVEFSLA, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.757979276 (497h13m18.629062321s) [noise=true , address=77.90.31.21:4878 , piece_id=RNZ3EK2ZERZY6IQEBAZPHEUDBHNF7BARJF5FVZMLVVMLH5OK7WCQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.758365802 (497h13m18.629448837s) [noise=true , address=45.91.116.4:4937 , piece_id=7HT3WA63RZNVO3AX775OL37N4BJRXZCZ5P6A2A2Z6NPZX6RASSSQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.758773228 (497h13m18.629856263s) [noise=true , address=77.90.31.10:4517 , piece_id=RD3ARVUTAOQRS3YUQ7WL2LF47CXQFSVDDJSJUAEZAFVGUU4HJ7KQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.759342462 (497h13m18.630425507s) [noise=true , address=145.79.227.37:4567 , piece_id=J7Z3543ZQ2TIGFFHBO7V7GJKTGOY54Z6WAPYDI7ALYRYDS6NEK2Q, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.760026635 (497h13m18.63110968s) [noise=true , address=145.79.227.48:4301 , piece_id=IS2W2WAK7NOMKFEHY3QK7KS35XPQYC2723Q2ZJ27HTX4FPBQYHDQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.760798124 (497h13m18.631881169s) [noise=true , address=145.79.227.51:4411 , piece_id=7JDMY4F74SKL6NSFN2HO36EGUCM5MWVWR5HLXACHBLZXTDRRWI3Q, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.761677319 (497h13m18.632760364s) [noise=true , address=77.90.31.14:4664 , piece_id=JFEBDNYO2OQRCKCLBKWS43SZCZ7L2LSIF5FP2JS37FFARUEHCESA, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:20.466851608 (497h13m20.337934643s) [upload=436400]: Next returned nil segment
Dec 26 08:42:20.466865273 (497h13m20.337948318s) [upload=436400]: Waiting for error group managing segments...
Dec 26 08:42:20.750402218 (497h13m20.621485253s) [noise=true , address=145.79.227.37:4567 , piece_id=J7Z3543ZQ2TIGFFHBO7V7GJKTGOY54Z6WAPYDI7ALYRYDS6NEK2Q, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:20.763579740 (497h13m20.634662785s) [noise=true , address=145.79.227.48:4301 , piece_id=IS2W2WAK7NOMKFEHY3QK7KS35XPQYC2723Q2ZJ27HTX4FPBQYHDQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:20.767844120 (497h13m20.638927155s) [noise=true , address=145.79.227.38:4977 , piece_id=MYMT3XBOXWU5DZOTPVHSVVXOKNHW7WWVCBECIO37VAYUTFK7XZSQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:20.771941653 (497h13m20.643024698s) [noise=true , address=145.79.227.16:4912 , piece_id=TIPMXPQDX4HVH2NUQ3FUFT6J34SNM5DT2NHDXCZO76MMGPEEEHMQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:20.773887688 (497h13m20.644970723s) [noise=true , address=145.79.227.17:4970 , piece_id=JXWYFTTKYNB6BSPBKWBK4YPF2PNAGWJKXHJAOUFIJVVIKIVNPUJA, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:20.781414641 (497h13m20.652497676s) [noise=true , address=145.79.227.51:4411 , piece_id=7JDMY4F74SKL6NSFN2HO36EGUCM5MWVWR5HLXACHBLZXTDRRWI3Q, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:20.781591407 (497h13m20.652674452s) [noise=true , address=145.79.227.46:4603 , piece_id=7L43455SH4JVSXBJYL3WVMRCWFQZ2CEI6DQA7MUXDYSKPPKESY7A, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:20.787415879 (497h13m20.658498924s) [noise=true , address=145.79.227.45:4229 , piece_id=GNXPJKQLV6ITICMBLE4PY2ODOKEJ6WWBUAPA74AO4IBVOVVEFSLA, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:21.029225186 (497h13m20.900308221s) [noise=true , address=45.91.116.4:4937 , piece_id=7HT3WA63RZNVO3AX775OL37N4BJRXZCZ5P6A2A2Z6NPZX6RASSSQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:21.124342636 (497h13m20.995425671s) [noise=true , address=77.90.31.21:4878 , piece_id=RNZ3EK2ZERZY6IQEBAZPHEUDBHNF7BARJF5FVZMLVVMLH5OK7WCQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:21.148939092 (497h13m21.020022127s) [noise=true , address=77.90.31.14:4664 , piece_id=JFEBDNYO2OQRCKCLBKWS43SZCZ7L2LSIF5FP2JS37FFARUEHCESA, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:21.733228030 (497h13m21.604311075s) [noise=true , address=77.90.31.11:4555 , piece_id=ZTSV63NMIHDFU464BAUD3MH3645MJOCGLTDQEKIRPD47EMLLO5EA, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:21.818455067 (497h13m21.689538102s) [noise=true , address=77.90.31.17:4773 , piece_id=P645R4U4WSCWJR5XZSGFGVVF3QHAVSP6SN46CPQGBT6JWJB57ZIQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:21.818474194 (497h13m21.689557239s) [seg_pos={0 0}, upload=436400]: Segment reached optimal threshold of 13 pieces.
Dec 26 08:42:21.818682150 (497h13m21.689765195s) [noise=true , address=77.90.31.10:4517 , piece_id=RD3ARVUTAOQRS3YUQ7WL2LF47CXQFSVDDJSJUAEZAFVGUU4HJ7KQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF
Dec 26 08:42:21.818690926 (497h13m21.689773971s) [noise=true , address=77.90.31.10:4517 , piece_id=RD3ARVUTAOQRS3YUQ7WL2LF47CXQFSVDDJSJUAEZAFVGUU4HJ7KQ, seg_pos={0 0}, upload=436400]: Long tail cancellation due to optimal threshold
Dec 26 08:42:21.818911596 (497h13m21.689994631s) [noise=true , address=145.79.227.40:4051 , piece_id=NLPGNO63SJLYA65HYN2CJBJFS6OTLX7ISULHSM6Q54PWOOTVAVWQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF
Dec 26 08:42:21.818918088 (497h13m21.690001123s) [noise=true , address=145.79.227.40:4051 , piece_id=NLPGNO63SJLYA65HYN2CJBJFS6OTLX7ISULHSM6Q54PWOOTVAVWQ, seg_pos={0 0}, upload=436400]: Long tail cancellation due to optimal threshold
Dec 26 08:42:21.818941363 (497h13m21.690024418s) [noise=true , address=77.90.31.18:4808 , piece_id=INW543LSVQLUQQ4FZZXVTVPV2LSGPKMUVHCSKE3OQ2CQAIEQFD2Q, seg_pos={0 0}, upload=436400]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: expected piece hash; context canceled; EOF
Dec 26 08:42:21.818997390 (497h13m21.690080425s) [noise=true , address=77.90.31.18:4808 , piece_id=INW543LSVQLUQQ4FZZXVTVPV2LSGPKMUVHCSKE3OQ2CQAIEQFD2Q, seg_pos={0 0}, upload=436400]: Long tail cancellation due to optimal threshold
Dec 26 08:42:21.819032707 (497h13m21.690115742s) [seg_pos={0 0}, upload=436400]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 26 08:42:21.819224622 (497h13m21.690307657s) [upload=436400]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 08:42:21.819232357 (497h13m21.690315392s) [upload=436400]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 08:42:22.109308312 (497h13m21.980391357s) [upload=436400]: Done uploading segments...
Dec 26 08:42:22.109331417 (497h13m21.980414452s) [upload=436400]: Upload finished. err: <nil>
Duration: 5.981685004s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [....................................X.......................] duration:
26.1980593s trace:
https://jaeger.dev.storj.io/trace/6820217d73ecf2a
FAIL (2025-12-26 08:52:13 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
ss=45.91.116.13:4174 , piece_id=RLKVQ3QJM4STACAOUTWA7STEIG4PITJRIB4YOI3GNTYMPA76IFHQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.705106031 (497h3m36.576189076s) [noise=true , address=77.90.31.19:4870 , piece_id=ZHJNUA7P36RBWXQXMDLFRIHQPSZNOPGL67AQVDTDEPCRY2R4EMWQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.705493759 (497h3m36.576576804s) [noise=true , address=145.79.227.20:4443 , piece_id=HAFVT75GMRHFRR5E5VD4AOJFHGRUS56MQX3HNHI6FQYYMZT4J4FQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.705630279 (497h3m36.576713334s) [noise=true , address=77.90.31.15:4717 , piece_id=CVL5BLBIJ5DWSL7ZSEJ4I62RENOFPNWWNZCSE62CTMWGSS2XTE2Q, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.705923348 (497h3m36.577006393s) [noise=true , address=77.90.31.12:4607 , piece_id=TPFAMNN2BEYPSYRYOMDMSS77ZRB35HOJYWMJGTJEERRWGV5KQHHA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706055580 (497h3m36.577138615s) [noise=true , address=64.49.13.169:4705 , piece_id=NG53QDRTA5IAV5W64W6MSZVNCWDUDHBB3DP4BF6G7DTF5DAF7ECA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706362764 (497h3m36.577445819s) [noise=true , address=77.90.31.11:4581 , piece_id=NQOGQYJLH7W7CXFLBDW3XUN6Y6YVQEWAFBMLNKNHKTU5VCV4V3GQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706507651 (497h3m36.577590686s) [noise=true , address=145.79.227.44:4226 , piece_id=XFE2F2254TY5CMO7J5O6S3RDKHOHSLQCUGDNHB37VKOXPM63GJEA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706728992 (497h3m36.577812027s) [noise=true , address=45.91.116.4:4942 , piece_id=VDKHG53ATFYJDQYDBXKZ7SN2RNAAZBYGQVGAF34PCG4UYO7XMFGA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706869399 (497h3m36.577952444s) [noise=true , address=77.90.31.16:4750 , piece_id=HIZWD4OHAZ6UKMODPFHABQRQW7ESEZPVMSD45RI5HGXPLHDJLJHQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707077395 (497h3m36.57816044s) [noise=true , address=145.79.227.34:4909 , piece_id=TSBSOK42K3BBLLTUBV6UFWW33FK67FZHJZTKEQFAAQ7ITKY3RAEQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707229524 (497h3m36.578312559s) [noise=true , address=77.90.31.13:4655 , piece_id=B6QS3X24QQRPWA3O5ENXR32XJOJZBZZOVVDMSESJYKU3XBIPGDIQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707443251 (497h3m36.578526296s) [noise=true , address=145.79.227.5:4241 , piece_id=ANDNWFXV5YWOH55WDLOMJXSKZVN45XSIYT2I4QB52DNO2PT3HYKQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707633815 (497h3m36.57871685s) [noise=true , address=64.49.13.154:4930 , piece_id=IHQYIUPLXMAKL4S76Y6NAGDP2ZWFRQBOXXJJENZ6JRGBXWLKDCFQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707813797 (497h3m36.578896832s) [noise=true , address=145.79.227.13:4822 , piece_id=HSHVLD6IYQOG3KQYCZGENM4ZYH7XCKGRAZLEL6N2WELIBI2LNHSA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.708022374 (497h3m36.579105439s) [noise=true , address=145.79.227.30:4775 , piece_id=OG65SPMQI3XHNJHUHKSJ4NZKGAB764OYFE7D5CBJCP3YRAGYANQA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.708682712 (497h3m36.579765747s) [noise=true , address=77.90.31.17:4788 , piece_id=CIK5XZI3QMG7LFEOKBPZGNLDOORNP6IQV3QYSHLIY5XSF3LLXBBQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:38.013251430 (497h3m37.884334475s) [upload=436256]: Next returned nil segment
Dec 26 08:32:38.013277480 (497h3m37.884360525s) [upload=436256]: Waiting for error group managing segments...
Dec 26 08:32:38.345076384 (497h3m38.216159419s) [noise=true , address=64.49.13.154:4930 , piece_id=IHQYIUPLXMAKL4S76Y6NAGDP2ZWFRQBOXXJJENZ6JRGBXWLKDCFQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.593234781 (497h3m38.464317836s) [noise=true , address=145.79.227.20:4443 , piece_id=HAFVT75GMRHFRR5E5VD4AOJFHGRUS56MQX3HNHI6FQYYMZT4J4FQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.593901300 (497h3m38.464984335s) [noise=true , address=145.79.227.5:4241 , piece_id=ANDNWFXV5YWOH55WDLOMJXSKZVN45XSIYT2I4QB52DNO2PT3HYKQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.594508065 (497h3m38.4655911s) [noise=true , address=145.79.227.13:4822 , piece_id=HSHVLD6IYQOG3KQYCZGENM4ZYH7XCKGRAZLEL6N2WELIBI2LNHSA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.636070587 (497h3m38.507153653s) [noise=true , address=145.79.227.34:4909 , piece_id=TSBSOK42K3BBLLTUBV6UFWW33FK67FZHJZTKEQFAAQ7ITKY3RAEQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.750629848 (497h3m38.621712883s) [noise=true , address=77.90.31.11:4581 , piece_id=NQOGQYJLH7W7CXFLBDW3XUN6Y6YVQEWAFBMLNKNHKTU5VCV4V3GQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.814987737 (497h3m38.686070782s) [noise=true , address=45.91.116.4:4942 , piece_id=VDKHG53ATFYJDQYDBXKZ7SN2RNAAZBYGQVGAF34PCG4UYO7XMFGA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.847327759 (497h3m38.718410794s) [noise=true , address=77.90.31.15:4717 , piece_id=CVL5BLBIJ5DWSL7ZSEJ4I62RENOFPNWWNZCSE62CTMWGSS2XTE2Q, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.902089014 (497h3m38.773172049s) [noise=true , address=45.91.116.13:4174 , piece_id=RLKVQ3QJM4STACAOUTWA7STEIG4PITJRIB4YOI3GNTYMPA76IFHQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.912841608 (497h3m38.783924644s) [noise=true , address=77.90.31.12:4607 , piece_id=TPFAMNN2BEYPSYRYOMDMSS77ZRB35HOJYWMJGTJEERRWGV5KQHHA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.070851326 (497h3m38.941934371s) [noise=true , address=77.90.31.16:4750 , piece_id=HIZWD4OHAZ6UKMODPFHABQRQW7ESEZPVMSD45RI5HGXPLHDJLJHQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.469018898 (497h3m39.340101933s) [noise=true , address=145.79.227.30:4775 , piece_id=OG65SPMQI3XHNJHUHKSJ4NZKGAB764OYFE7D5CBJCP3YRAGYANQA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.477948602 (497h3m39.349031637s) [noise=true , address=145.79.227.44:4226 , piece_id=XFE2F2254TY5CMO7J5O6S3RDKHOHSLQCUGDNHB37VKOXPM63GJEA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.534878005 (497h3m39.40596105s) [noise=true , address=77.90.31.17:4788 , piece_id=CIK5XZI3QMG7LFEOKBPZGNLDOORNP6IQV3QYSHLIY5XSF3LLXBBQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.534902853 (497h3m39.405985898s) [seg_pos={0 0}, upload=436256]: Segment reached optimal threshold of 14 pieces.
Dec 26 08:32:39.535131698 (497h3m39.406214743s) [noise=true , address=77.90.31.19:4870 , piece_id=ZHJNUA7P36RBWXQXMDLFRIHQPSZNOPGL67AQVDTDEPCRY2R4EMWQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hNVXEVe8DW8F8meERGWkxAazEZKLKBeprNn6e5C94eEoAMBaR): protocol: expected piece hash; context canceled; EOF
Dec 26 08:32:39.535140075 (497h3m39.40622311s) [noise=true , address=77.90.31.19:4870 , piece_id=ZHJNUA7P36RBWXQXMDLFRIHQPSZNOPGL67AQVDTDEPCRY2R4EMWQ, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold
Dec 26 08:32:39.535367978 (497h3m39.406451013s) [noise=true , address=77.90.31.18:4838 , piece_id=V5LDUMURFU4H43E2A5MTLWBI6ZJHVPYMUILBDV5N7ERELPLTIBAA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1if7vf1cNTBwAhb6owi6tuGLYqSfa3feWhsvGSevwuwykhz2RY): protocol: expected piece hash; context canceled; EOF
Dec 26 08:32:39.535375542 (497h3m39.406458577s) [noise=true , address=77.90.31.18:4838 , piece_id=V5LDUMURFU4H43E2A5MTLWBI6ZJHVPYMUILBDV5N7ERELPLTIBAA, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold
Dec 26 08:32:39.535565575 (497h3m39.40664861s) [noise=true , address=77.90.31.13:4655 , piece_id=B6QS3X24QQRPWA3O5ENXR32XJOJZBZZOVVDMSESJYKU3XBIPGDIQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1K7fQM1NVA1L7ukmDtyWh2DYWypUoj3GbreETZpPL6gXfXmZDD): protocol: expected piece hash; context canceled; EOF
Dec 26 08:32:39.535606041 (497h3m39.406689077s) [noise=true , address=77.90.31.13:4655 , piece_id=B6QS3X24QQRPWA3O5ENXR32XJOJZBZZOVVDMSESJYKU3XBIPGDIQ, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold
Dec 26 08:32:39.535707996 (497h3m39.406791031s) [noise=true , address=64.49.13.169:4705 , piece_id=NG53QDRTA5IAV5W64W6MSZVNCWDUDHBB3DP4BF6G7DTF5DAF7ECA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: read tcp 188.166.225.170:35726->64.49.13.169:4705: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 26 08:32:39.535715059 (497h3m39.406798094s) [noise=true , address=64.49.13.169:4705 , piece_id=NG53QDRTA5IAV5W64W6MSZVNCWDUDHBB3DP4BF6G7DTF5DAF7ECA, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold
Dec 26 08:32:39.535740878 (497h3m39.406823923s) [seg_pos={0 0}, upload=436256]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 26 08:32:39.535861157 (497h3m39.406944202s) [upload=436256]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 08:32:39.535867620 (497h3m39.406950655s) [upload=436256]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 08:32:39.833497059 (497h3m39.704580164s) [upload=436256]: Done uploading segments...
Dec 26 08:32:39.833522858 (497h3m39.704605903s) [upload=436256]: Upload finished. err: <nil>
Duration: 26.198078838s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [.X..........................................................] duration:
3.498992964s trace:
https://jaeger.dev.storj.io/trace/185fddc4dc40eba9
FAIL (2025-12-26 08:52:09 UTC)
orbiter warning: context deadline exceeded
Duration: 3.499031817s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..X.........................................................] duration:
6.501677187s trace:
https://jaeger.dev.storj.io/trace/4b8b74dd4906ed4
FAIL (2025-12-26 08:51:39 UTC)
orbiter warning: context deadline exceeded
Duration: 6.50170474s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [..X.........................................................] duration:
3.46535005s trace:
https://jaeger.dev.storj.io/trace/185fddc4dc40eba9
FAIL (2025-12-26 08:51:36 UTC)
orbiter warning: context deadline exceeded
Duration: 3.465377193s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.....................................X......................] duration:
45.188485623s trace:
https://jaeger.dev.storj.io/trace/6820217d73ecf2a
FAIL (2025-12-26 08:51:28 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
ss=45.91.116.13:4174 , piece_id=RLKVQ3QJM4STACAOUTWA7STEIG4PITJRIB4YOI3GNTYMPA76IFHQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.705106031 (497h3m36.576189076s) [noise=true , address=77.90.31.19:4870 , piece_id=ZHJNUA7P36RBWXQXMDLFRIHQPSZNOPGL67AQVDTDEPCRY2R4EMWQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.705493759 (497h3m36.576576804s) [noise=true , address=145.79.227.20:4443 , piece_id=HAFVT75GMRHFRR5E5VD4AOJFHGRUS56MQX3HNHI6FQYYMZT4J4FQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.705630279 (497h3m36.576713334s) [noise=true , address=77.90.31.15:4717 , piece_id=CVL5BLBIJ5DWSL7ZSEJ4I62RENOFPNWWNZCSE62CTMWGSS2XTE2Q, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.705923348 (497h3m36.577006393s) [noise=true , address=77.90.31.12:4607 , piece_id=TPFAMNN2BEYPSYRYOMDMSS77ZRB35HOJYWMJGTJEERRWGV5KQHHA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706055580 (497h3m36.577138615s) [noise=true , address=64.49.13.169:4705 , piece_id=NG53QDRTA5IAV5W64W6MSZVNCWDUDHBB3DP4BF6G7DTF5DAF7ECA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706362764 (497h3m36.577445819s) [noise=true , address=77.90.31.11:4581 , piece_id=NQOGQYJLH7W7CXFLBDW3XUN6Y6YVQEWAFBMLNKNHKTU5VCV4V3GQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706507651 (497h3m36.577590686s) [noise=true , address=145.79.227.44:4226 , piece_id=XFE2F2254TY5CMO7J5O6S3RDKHOHSLQCUGDNHB37VKOXPM63GJEA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706728992 (497h3m36.577812027s) [noise=true , address=45.91.116.4:4942 , piece_id=VDKHG53ATFYJDQYDBXKZ7SN2RNAAZBYGQVGAF34PCG4UYO7XMFGA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706869399 (497h3m36.577952444s) [noise=true , address=77.90.31.16:4750 , piece_id=HIZWD4OHAZ6UKMODPFHABQRQW7ESEZPVMSD45RI5HGXPLHDJLJHQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707077395 (497h3m36.57816044s) [noise=true , address=145.79.227.34:4909 , piece_id=TSBSOK42K3BBLLTUBV6UFWW33FK67FZHJZTKEQFAAQ7ITKY3RAEQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707229524 (497h3m36.578312559s) [noise=true , address=77.90.31.13:4655 , piece_id=B6QS3X24QQRPWA3O5ENXR32XJOJZBZZOVVDMSESJYKU3XBIPGDIQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707443251 (497h3m36.578526296s) [noise=true , address=145.79.227.5:4241 , piece_id=ANDNWFXV5YWOH55WDLOMJXSKZVN45XSIYT2I4QB52DNO2PT3HYKQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707633815 (497h3m36.57871685s) [noise=true , address=64.49.13.154:4930 , piece_id=IHQYIUPLXMAKL4S76Y6NAGDP2ZWFRQBOXXJJENZ6JRGBXWLKDCFQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707813797 (497h3m36.578896832s) [noise=true , address=145.79.227.13:4822 , piece_id=HSHVLD6IYQOG3KQYCZGENM4ZYH7XCKGRAZLEL6N2WELIBI2LNHSA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.708022374 (497h3m36.579105439s) [noise=true , address=145.79.227.30:4775 , piece_id=OG65SPMQI3XHNJHUHKSJ4NZKGAB764OYFE7D5CBJCP3YRAGYANQA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.708682712 (497h3m36.579765747s) [noise=true , address=77.90.31.17:4788 , piece_id=CIK5XZI3QMG7LFEOKBPZGNLDOORNP6IQV3QYSHLIY5XSF3LLXBBQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:38.013251430 (497h3m37.884334475s) [upload=436256]: Next returned nil segment
Dec 26 08:32:38.013277480 (497h3m37.884360525s) [upload=436256]: Waiting for error group managing segments...
Dec 26 08:32:38.345076384 (497h3m38.216159419s) [noise=true , address=64.49.13.154:4930 , piece_id=IHQYIUPLXMAKL4S76Y6NAGDP2ZWFRQBOXXJJENZ6JRGBXWLKDCFQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.593234781 (497h3m38.464317836s) [noise=true , address=145.79.227.20:4443 , piece_id=HAFVT75GMRHFRR5E5VD4AOJFHGRUS56MQX3HNHI6FQYYMZT4J4FQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.593901300 (497h3m38.464984335s) [noise=true , address=145.79.227.5:4241 , piece_id=ANDNWFXV5YWOH55WDLOMJXSKZVN45XSIYT2I4QB52DNO2PT3HYKQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.594508065 (497h3m38.4655911s) [noise=true , address=145.79.227.13:4822 , piece_id=HSHVLD6IYQOG3KQYCZGENM4ZYH7XCKGRAZLEL6N2WELIBI2LNHSA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.636070587 (497h3m38.507153653s) [noise=true , address=145.79.227.34:4909 , piece_id=TSBSOK42K3BBLLTUBV6UFWW33FK67FZHJZTKEQFAAQ7ITKY3RAEQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.750629848 (497h3m38.621712883s) [noise=true , address=77.90.31.11:4581 , piece_id=NQOGQYJLH7W7CXFLBDW3XUN6Y6YVQEWAFBMLNKNHKTU5VCV4V3GQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.814987737 (497h3m38.686070782s) [noise=true , address=45.91.116.4:4942 , piece_id=VDKHG53ATFYJDQYDBXKZ7SN2RNAAZBYGQVGAF34PCG4UYO7XMFGA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.847327759 (497h3m38.718410794s) [noise=true , address=77.90.31.15:4717 , piece_id=CVL5BLBIJ5DWSL7ZSEJ4I62RENOFPNWWNZCSE62CTMWGSS2XTE2Q, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.902089014 (497h3m38.773172049s) [noise=true , address=45.91.116.13:4174 , piece_id=RLKVQ3QJM4STACAOUTWA7STEIG4PITJRIB4YOI3GNTYMPA76IFHQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.912841608 (497h3m38.783924644s) [noise=true , address=77.90.31.12:4607 , piece_id=TPFAMNN2BEYPSYRYOMDMSS77ZRB35HOJYWMJGTJEERRWGV5KQHHA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.070851326 (497h3m38.941934371s) [noise=true , address=77.90.31.16:4750 , piece_id=HIZWD4OHAZ6UKMODPFHABQRQW7ESEZPVMSD45RI5HGXPLHDJLJHQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.469018898 (497h3m39.340101933s) [noise=true , address=145.79.227.30:4775 , piece_id=OG65SPMQI3XHNJHUHKSJ4NZKGAB764OYFE7D5CBJCP3YRAGYANQA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.477948602 (497h3m39.349031637s) [noise=true , address=145.79.227.44:4226 , piece_id=XFE2F2254TY5CMO7J5O6S3RDKHOHSLQCUGDNHB37VKOXPM63GJEA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.534878005 (497h3m39.40596105s) [noise=true , address=77.90.31.17:4788 , piece_id=CIK5XZI3QMG7LFEOKBPZGNLDOORNP6IQV3QYSHLIY5XSF3LLXBBQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.534902853 (497h3m39.405985898s) [seg_pos={0 0}, upload=436256]: Segment reached optimal threshold of 14 pieces.
Dec 26 08:32:39.535131698 (497h3m39.406214743s) [noise=true , address=77.90.31.19:4870 , piece_id=ZHJNUA7P36RBWXQXMDLFRIHQPSZNOPGL67AQVDTDEPCRY2R4EMWQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hNVXEVe8DW8F8meERGWkxAazEZKLKBeprNn6e5C94eEoAMBaR): protocol: expected piece hash; context canceled; EOF
Dec 26 08:32:39.535140075 (497h3m39.40622311s) [noise=true , address=77.90.31.19:4870 , piece_id=ZHJNUA7P36RBWXQXMDLFRIHQPSZNOPGL67AQVDTDEPCRY2R4EMWQ, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold
Dec 26 08:32:39.535367978 (497h3m39.406451013s) [noise=true , address=77.90.31.18:4838 , piece_id=V5LDUMURFU4H43E2A5MTLWBI6ZJHVPYMUILBDV5N7ERELPLTIBAA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1if7vf1cNTBwAhb6owi6tuGLYqSfa3feWhsvGSevwuwykhz2RY): protocol: expected piece hash; context canceled; EOF
Dec 26 08:32:39.535375542 (497h3m39.406458577s) [noise=true , address=77.90.31.18:4838 , piece_id=V5LDUMURFU4H43E2A5MTLWBI6ZJHVPYMUILBDV5N7ERELPLTIBAA, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold
Dec 26 08:32:39.535565575 (497h3m39.40664861s) [noise=true , address=77.90.31.13:4655 , piece_id=B6QS3X24QQRPWA3O5ENXR32XJOJZBZZOVVDMSESJYKU3XBIPGDIQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1K7fQM1NVA1L7ukmDtyWh2DYWypUoj3GbreETZpPL6gXfXmZDD): protocol: expected piece hash; context canceled; EOF
Dec 26 08:32:39.535606041 (497h3m39.406689077s) [noise=true , address=77.90.31.13:4655 , piece_id=B6QS3X24QQRPWA3O5ENXR32XJOJZBZZOVVDMSESJYKU3XBIPGDIQ, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold
Dec 26 08:32:39.535707996 (497h3m39.406791031s) [noise=true , address=64.49.13.169:4705 , piece_id=NG53QDRTA5IAV5W64W6MSZVNCWDUDHBB3DP4BF6G7DTF5DAF7ECA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: read tcp 188.166.225.170:35726->64.49.13.169:4705: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 26 08:32:39.535715059 (497h3m39.406798094s) [noise=true , address=64.49.13.169:4705 , piece_id=NG53QDRTA5IAV5W64W6MSZVNCWDUDHBB3DP4BF6G7DTF5DAF7ECA, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold
Dec 26 08:32:39.535740878 (497h3m39.406823923s) [seg_pos={0 0}, upload=436256]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 26 08:32:39.535861157 (497h3m39.406944202s) [upload=436256]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 08:32:39.535867620 (497h3m39.406950655s) [upload=436256]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 08:32:39.833497059 (497h3m39.704580164s) [upload=436256]: Done uploading segments...
Dec 26 08:32:39.833522858 (497h3m39.704605903s) [upload=436256]: Upload finished. err: <nil>
Duration: 45.188504369s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
20 out of 60 is failed (threshold=3) [..X.XX.X.X..X..X.XXX.....X......X.X...X.XX..X.XXX...........] duration:
6.146736861s trace:
https://jaeger.dev.storj.io/trace/3312cc3668ef2c5f
FAIL (2025-12-26 08:51:09 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 26 08:42:18.506798665 (497h13m18.37788171s) [upload=436400]: Starting upload
Dec 26 08:42:18.506881102 (497h13m18.377964137s) [upload=436400]: Done starting upload
Dec 26 08:42:18.506963669 (497h13m18.378046714s) [upload=436400]: Uploading segments...
Dec 26 08:42:18.507170413 (497h13m18.378253478s) [upload=436400]: Got next segment. Inline: false
Dec 26 08:42:18.507247920 (497h13m18.378330965s) [upload=436400]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 26 08:42:18.507255454 (497h13m18.378338499s) [upload=436400]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 26 08:42:18.754283459 (497h13m18.625366504s) [seg_pos={0 0}, upload=436400]: Begin upload segment...
Dec 26 08:42:18.754356328 (497h13m18.625439373s) [seg_pos={0 0}, upload=436400]: Done begin upload segment.
Dec 26 08:42:18.754391745 (497h13m18.62547479s) [noise=true , address=145.79.227.40:4051 , piece_id=NLPGNO63SJLYA65HYN2CJBJFS6OTLX7ISULHSM6Q54PWOOTVAVWQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.754466177 (497h13m18.625549222s) [noise=true , address=77.90.31.17:4773 , piece_id=P645R4U4WSCWJR5XZSGFGVVF3QHAVSP6SN46CPQGBT6JWJB57ZIQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.755567554 (497h13m18.626650599s) [noise=true , address=77.90.31.18:4808 , piece_id=INW543LSVQLUQQ4FZZXVTVPV2LSGPKMUVHCSKE3OQ2CQAIEQFD2Q, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.755910858 (497h13m18.626993893s) [noise=true , address=145.79.227.17:4970 , piece_id=JXWYFTTKYNB6BSPBKWBK4YPF2PNAGWJKXHJAOUFIJVVIKIVNPUJA, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.756229915 (497h13m18.62731296s) [noise=true , address=145.79.227.46:4603 , piece_id=7L43455SH4JVSXBJYL3WVMRCWFQZ2CEI6DQA7MUXDYSKPPKESY7A, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.756613886 (497h13m18.627696922s) [noise=true , address=145.79.227.16:4912 , piece_id=TIPMXPQDX4HVH2NUQ3FUFT6J34SNM5DT2NHDXCZO76MMGPEEEHMQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.757020902 (497h13m18.628103937s) [noise=true , address=145.79.227.38:4977 , piece_id=MYMT3XBOXWU5DZOTPVHSVVXOKNHW7WWVCBECIO37VAYUTFK7XZSQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.757428849 (497h13m18.628511904s) [noise=true , address=77.90.31.11:4555 , piece_id=ZTSV63NMIHDFU464BAUD3MH3645MJOCGLTDQEKIRPD47EMLLO5EA, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.757902380 (497h13m18.628985425s) [noise=true , address=145.79.227.45:4229 , piece_id=GNXPJKQLV6ITICMBLE4PY2ODOKEJ6WWBUAPA74AO4IBVOVVEFSLA, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.757979276 (497h13m18.629062321s) [noise=true , address=77.90.31.21:4878 , piece_id=RNZ3EK2ZERZY6IQEBAZPHEUDBHNF7BARJF5FVZMLVVMLH5OK7WCQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.758365802 (497h13m18.629448837s) [noise=true , address=45.91.116.4:4937 , piece_id=7HT3WA63RZNVO3AX775OL37N4BJRXZCZ5P6A2A2Z6NPZX6RASSSQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.758773228 (497h13m18.629856263s) [noise=true , address=77.90.31.10:4517 , piece_id=RD3ARVUTAOQRS3YUQ7WL2LF47CXQFSVDDJSJUAEZAFVGUU4HJ7KQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.759342462 (497h13m18.630425507s) [noise=true , address=145.79.227.37:4567 , piece_id=J7Z3543ZQ2TIGFFHBO7V7GJKTGOY54Z6WAPYDI7ALYRYDS6NEK2Q, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.760026635 (497h13m18.63110968s) [noise=true , address=145.79.227.48:4301 , piece_id=IS2W2WAK7NOMKFEHY3QK7KS35XPQYC2723Q2ZJ27HTX4FPBQYHDQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.760798124 (497h13m18.631881169s) [noise=true , address=145.79.227.51:4411 , piece_id=7JDMY4F74SKL6NSFN2HO36EGUCM5MWVWR5HLXACHBLZXTDRRWI3Q, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.761677319 (497h13m18.632760364s) [noise=true , address=77.90.31.14:4664 , piece_id=JFEBDNYO2OQRCKCLBKWS43SZCZ7L2LSIF5FP2JS37FFARUEHCESA, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:20.466851608 (497h13m20.337934643s) [upload=436400]: Next returned nil segment
Dec 26 08:42:20.466865273 (497h13m20.337948318s) [upload=436400]: Waiting for error group managing segments...
Dec 26 08:42:20.750402218 (497h13m20.621485253s) [noise=true , address=145.79.227.37:4567 , piece_id=J7Z3543ZQ2TIGFFHBO7V7GJKTGOY54Z6WAPYDI7ALYRYDS6NEK2Q, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:20.763579740 (497h13m20.634662785s) [noise=true , address=145.79.227.48:4301 , piece_id=IS2W2WAK7NOMKFEHY3QK7KS35XPQYC2723Q2ZJ27HTX4FPBQYHDQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:20.767844120 (497h13m20.638927155s) [noise=true , address=145.79.227.38:4977 , piece_id=MYMT3XBOXWU5DZOTPVHSVVXOKNHW7WWVCBECIO37VAYUTFK7XZSQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:20.771941653 (497h13m20.643024698s) [noise=true , address=145.79.227.16:4912 , piece_id=TIPMXPQDX4HVH2NUQ3FUFT6J34SNM5DT2NHDXCZO76MMGPEEEHMQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:20.773887688 (497h13m20.644970723s) [noise=true , address=145.79.227.17:4970 , piece_id=JXWYFTTKYNB6BSPBKWBK4YPF2PNAGWJKXHJAOUFIJVVIKIVNPUJA, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:20.781414641 (497h13m20.652497676s) [noise=true , address=145.79.227.51:4411 , piece_id=7JDMY4F74SKL6NSFN2HO36EGUCM5MWVWR5HLXACHBLZXTDRRWI3Q, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:20.781591407 (497h13m20.652674452s) [noise=true , address=145.79.227.46:4603 , piece_id=7L43455SH4JVSXBJYL3WVMRCWFQZ2CEI6DQA7MUXDYSKPPKESY7A, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:20.787415879 (497h13m20.658498924s) [noise=true , address=145.79.227.45:4229 , piece_id=GNXPJKQLV6ITICMBLE4PY2ODOKEJ6WWBUAPA74AO4IBVOVVEFSLA, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:21.029225186 (497h13m20.900308221s) [noise=true , address=45.91.116.4:4937 , piece_id=7HT3WA63RZNVO3AX775OL37N4BJRXZCZ5P6A2A2Z6NPZX6RASSSQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:21.124342636 (497h13m20.995425671s) [noise=true , address=77.90.31.21:4878 , piece_id=RNZ3EK2ZERZY6IQEBAZPHEUDBHNF7BARJF5FVZMLVVMLH5OK7WCQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:21.148939092 (497h13m21.020022127s) [noise=true , address=77.90.31.14:4664 , piece_id=JFEBDNYO2OQRCKCLBKWS43SZCZ7L2LSIF5FP2JS37FFARUEHCESA, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:21.733228030 (497h13m21.604311075s) [noise=true , address=77.90.31.11:4555 , piece_id=ZTSV63NMIHDFU464BAUD3MH3645MJOCGLTDQEKIRPD47EMLLO5EA, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:21.818455067 (497h13m21.689538102s) [noise=true , address=77.90.31.17:4773 , piece_id=P645R4U4WSCWJR5XZSGFGVVF3QHAVSP6SN46CPQGBT6JWJB57ZIQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:21.818474194 (497h13m21.689557239s) [seg_pos={0 0}, upload=436400]: Segment reached optimal threshold of 13 pieces.
Dec 26 08:42:21.818682150 (497h13m21.689765195s) [noise=true , address=77.90.31.10:4517 , piece_id=RD3ARVUTAOQRS3YUQ7WL2LF47CXQFSVDDJSJUAEZAFVGUU4HJ7KQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF
Dec 26 08:42:21.818690926 (497h13m21.689773971s) [noise=true , address=77.90.31.10:4517 , piece_id=RD3ARVUTAOQRS3YUQ7WL2LF47CXQFSVDDJSJUAEZAFVGUU4HJ7KQ, seg_pos={0 0}, upload=436400]: Long tail cancellation due to optimal threshold
Dec 26 08:42:21.818911596 (497h13m21.689994631s) [noise=true , address=145.79.227.40:4051 , piece_id=NLPGNO63SJLYA65HYN2CJBJFS6OTLX7ISULHSM6Q54PWOOTVAVWQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF
Dec 26 08:42:21.818918088 (497h13m21.690001123s) [noise=true , address=145.79.227.40:4051 , piece_id=NLPGNO63SJLYA65HYN2CJBJFS6OTLX7ISULHSM6Q54PWOOTVAVWQ, seg_pos={0 0}, upload=436400]: Long tail cancellation due to optimal threshold
Dec 26 08:42:21.818941363 (497h13m21.690024418s) [noise=true , address=77.90.31.18:4808 , piece_id=INW543LSVQLUQQ4FZZXVTVPV2LSGPKMUVHCSKE3OQ2CQAIEQFD2Q, seg_pos={0 0}, upload=436400]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: expected piece hash; context canceled; EOF
Dec 26 08:42:21.818997390 (497h13m21.690080425s) [noise=true , address=77.90.31.18:4808 , piece_id=INW543LSVQLUQQ4FZZXVTVPV2LSGPKMUVHCSKE3OQ2CQAIEQFD2Q, seg_pos={0 0}, upload=436400]: Long tail cancellation due to optimal threshold
Dec 26 08:42:21.819032707 (497h13m21.690115742s) [seg_pos={0 0}, upload=436400]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 26 08:42:21.819224622 (497h13m21.690307657s) [upload=436400]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 08:42:21.819232357 (497h13m21.690315392s) [upload=436400]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 08:42:22.109308312 (497h13m21.980391357s) [upload=436400]: Done uploading segments...
Dec 26 08:42:22.109331417 (497h13m21.980414452s) [upload=436400]: Upload finished. err: <nil>
Duration: 6.146759784s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...X........................................................] duration:
7.064192767s trace:
https://jaeger.dev.storj.io/trace/4b8b74dd4906ed4
FAIL (2025-12-26 08:50:48 UTC)
orbiter warning: context deadline exceeded
Duration: 7.064217073s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [......................................X.....................] duration:
49.451661453s trace:
https://jaeger.dev.storj.io/trace/6820217d73ecf2a
FAIL (2025-12-26 08:50:42 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
ss=45.91.116.13:4174 , piece_id=RLKVQ3QJM4STACAOUTWA7STEIG4PITJRIB4YOI3GNTYMPA76IFHQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.705106031 (497h3m36.576189076s) [noise=true , address=77.90.31.19:4870 , piece_id=ZHJNUA7P36RBWXQXMDLFRIHQPSZNOPGL67AQVDTDEPCRY2R4EMWQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.705493759 (497h3m36.576576804s) [noise=true , address=145.79.227.20:4443 , piece_id=HAFVT75GMRHFRR5E5VD4AOJFHGRUS56MQX3HNHI6FQYYMZT4J4FQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.705630279 (497h3m36.576713334s) [noise=true , address=77.90.31.15:4717 , piece_id=CVL5BLBIJ5DWSL7ZSEJ4I62RENOFPNWWNZCSE62CTMWGSS2XTE2Q, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.705923348 (497h3m36.577006393s) [noise=true , address=77.90.31.12:4607 , piece_id=TPFAMNN2BEYPSYRYOMDMSS77ZRB35HOJYWMJGTJEERRWGV5KQHHA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706055580 (497h3m36.577138615s) [noise=true , address=64.49.13.169:4705 , piece_id=NG53QDRTA5IAV5W64W6MSZVNCWDUDHBB3DP4BF6G7DTF5DAF7ECA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706362764 (497h3m36.577445819s) [noise=true , address=77.90.31.11:4581 , piece_id=NQOGQYJLH7W7CXFLBDW3XUN6Y6YVQEWAFBMLNKNHKTU5VCV4V3GQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706507651 (497h3m36.577590686s) [noise=true , address=145.79.227.44:4226 , piece_id=XFE2F2254TY5CMO7J5O6S3RDKHOHSLQCUGDNHB37VKOXPM63GJEA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706728992 (497h3m36.577812027s) [noise=true , address=45.91.116.4:4942 , piece_id=VDKHG53ATFYJDQYDBXKZ7SN2RNAAZBYGQVGAF34PCG4UYO7XMFGA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706869399 (497h3m36.577952444s) [noise=true , address=77.90.31.16:4750 , piece_id=HIZWD4OHAZ6UKMODPFHABQRQW7ESEZPVMSD45RI5HGXPLHDJLJHQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707077395 (497h3m36.57816044s) [noise=true , address=145.79.227.34:4909 , piece_id=TSBSOK42K3BBLLTUBV6UFWW33FK67FZHJZTKEQFAAQ7ITKY3RAEQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707229524 (497h3m36.578312559s) [noise=true , address=77.90.31.13:4655 , piece_id=B6QS3X24QQRPWA3O5ENXR32XJOJZBZZOVVDMSESJYKU3XBIPGDIQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707443251 (497h3m36.578526296s) [noise=true , address=145.79.227.5:4241 , piece_id=ANDNWFXV5YWOH55WDLOMJXSKZVN45XSIYT2I4QB52DNO2PT3HYKQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707633815 (497h3m36.57871685s) [noise=true , address=64.49.13.154:4930 , piece_id=IHQYIUPLXMAKL4S76Y6NAGDP2ZWFRQBOXXJJENZ6JRGBXWLKDCFQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707813797 (497h3m36.578896832s) [noise=true , address=145.79.227.13:4822 , piece_id=HSHVLD6IYQOG3KQYCZGENM4ZYH7XCKGRAZLEL6N2WELIBI2LNHSA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.708022374 (497h3m36.579105439s) [noise=true , address=145.79.227.30:4775 , piece_id=OG65SPMQI3XHNJHUHKSJ4NZKGAB764OYFE7D5CBJCP3YRAGYANQA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.708682712 (497h3m36.579765747s) [noise=true , address=77.90.31.17:4788 , piece_id=CIK5XZI3QMG7LFEOKBPZGNLDOORNP6IQV3QYSHLIY5XSF3LLXBBQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:38.013251430 (497h3m37.884334475s) [upload=436256]: Next returned nil segment
Dec 26 08:32:38.013277480 (497h3m37.884360525s) [upload=436256]: Waiting for error group managing segments...
Dec 26 08:32:38.345076384 (497h3m38.216159419s) [noise=true , address=64.49.13.154:4930 , piece_id=IHQYIUPLXMAKL4S76Y6NAGDP2ZWFRQBOXXJJENZ6JRGBXWLKDCFQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.593234781 (497h3m38.464317836s) [noise=true , address=145.79.227.20:4443 , piece_id=HAFVT75GMRHFRR5E5VD4AOJFHGRUS56MQX3HNHI6FQYYMZT4J4FQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.593901300 (497h3m38.464984335s) [noise=true , address=145.79.227.5:4241 , piece_id=ANDNWFXV5YWOH55WDLOMJXSKZVN45XSIYT2I4QB52DNO2PT3HYKQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.594508065 (497h3m38.4655911s) [noise=true , address=145.79.227.13:4822 , piece_id=HSHVLD6IYQOG3KQYCZGENM4ZYH7XCKGRAZLEL6N2WELIBI2LNHSA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.636070587 (497h3m38.507153653s) [noise=true , address=145.79.227.34:4909 , piece_id=TSBSOK42K3BBLLTUBV6UFWW33FK67FZHJZTKEQFAAQ7ITKY3RAEQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.750629848 (497h3m38.621712883s) [noise=true , address=77.90.31.11:4581 , piece_id=NQOGQYJLH7W7CXFLBDW3XUN6Y6YVQEWAFBMLNKNHKTU5VCV4V3GQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.814987737 (497h3m38.686070782s) [noise=true , address=45.91.116.4:4942 , piece_id=VDKHG53ATFYJDQYDBXKZ7SN2RNAAZBYGQVGAF34PCG4UYO7XMFGA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.847327759 (497h3m38.718410794s) [noise=true , address=77.90.31.15:4717 , piece_id=CVL5BLBIJ5DWSL7ZSEJ4I62RENOFPNWWNZCSE62CTMWGSS2XTE2Q, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.902089014 (497h3m38.773172049s) [noise=true , address=45.91.116.13:4174 , piece_id=RLKVQ3QJM4STACAOUTWA7STEIG4PITJRIB4YOI3GNTYMPA76IFHQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.912841608 (497h3m38.783924644s) [noise=true , address=77.90.31.12:4607 , piece_id=TPFAMNN2BEYPSYRYOMDMSS77ZRB35HOJYWMJGTJEERRWGV5KQHHA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.070851326 (497h3m38.941934371s) [noise=true , address=77.90.31.16:4750 , piece_id=HIZWD4OHAZ6UKMODPFHABQRQW7ESEZPVMSD45RI5HGXPLHDJLJHQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.469018898 (497h3m39.340101933s) [noise=true , address=145.79.227.30:4775 , piece_id=OG65SPMQI3XHNJHUHKSJ4NZKGAB764OYFE7D5CBJCP3YRAGYANQA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.477948602 (497h3m39.349031637s) [noise=true , address=145.79.227.44:4226 , piece_id=XFE2F2254TY5CMO7J5O6S3RDKHOHSLQCUGDNHB37VKOXPM63GJEA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.534878005 (497h3m39.40596105s) [noise=true , address=77.90.31.17:4788 , piece_id=CIK5XZI3QMG7LFEOKBPZGNLDOORNP6IQV3QYSHLIY5XSF3LLXBBQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.534902853 (497h3m39.405985898s) [seg_pos={0 0}, upload=436256]: Segment reached optimal threshold of 14 pieces.
Dec 26 08:32:39.535131698 (497h3m39.406214743s) [noise=true , address=77.90.31.19:4870 , piece_id=ZHJNUA7P36RBWXQXMDLFRIHQPSZNOPGL67AQVDTDEPCRY2R4EMWQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hNVXEVe8DW8F8meERGWkxAazEZKLKBeprNn6e5C94eEoAMBaR): protocol: expected piece hash; context canceled; EOF
Dec 26 08:32:39.535140075 (497h3m39.40622311s) [noise=true , address=77.90.31.19:4870 , piece_id=ZHJNUA7P36RBWXQXMDLFRIHQPSZNOPGL67AQVDTDEPCRY2R4EMWQ, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold
Dec 26 08:32:39.535367978 (497h3m39.406451013s) [noise=true , address=77.90.31.18:4838 , piece_id=V5LDUMURFU4H43E2A5MTLWBI6ZJHVPYMUILBDV5N7ERELPLTIBAA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1if7vf1cNTBwAhb6owi6tuGLYqSfa3feWhsvGSevwuwykhz2RY): protocol: expected piece hash; context canceled; EOF
Dec 26 08:32:39.535375542 (497h3m39.406458577s) [noise=true , address=77.90.31.18:4838 , piece_id=V5LDUMURFU4H43E2A5MTLWBI6ZJHVPYMUILBDV5N7ERELPLTIBAA, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold
Dec 26 08:32:39.535565575 (497h3m39.40664861s) [noise=true , address=77.90.31.13:4655 , piece_id=B6QS3X24QQRPWA3O5ENXR32XJOJZBZZOVVDMSESJYKU3XBIPGDIQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1K7fQM1NVA1L7ukmDtyWh2DYWypUoj3GbreETZpPL6gXfXmZDD): protocol: expected piece hash; context canceled; EOF
Dec 26 08:32:39.535606041 (497h3m39.406689077s) [noise=true , address=77.90.31.13:4655 , piece_id=B6QS3X24QQRPWA3O5ENXR32XJOJZBZZOVVDMSESJYKU3XBIPGDIQ, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold
Dec 26 08:32:39.535707996 (497h3m39.406791031s) [noise=true , address=64.49.13.169:4705 , piece_id=NG53QDRTA5IAV5W64W6MSZVNCWDUDHBB3DP4BF6G7DTF5DAF7ECA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: read tcp 188.166.225.170:35726->64.49.13.169:4705: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 26 08:32:39.535715059 (497h3m39.406798094s) [noise=true , address=64.49.13.169:4705 , piece_id=NG53QDRTA5IAV5W64W6MSZVNCWDUDHBB3DP4BF6G7DTF5DAF7ECA, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold
Dec 26 08:32:39.535740878 (497h3m39.406823923s) [seg_pos={0 0}, upload=436256]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 26 08:32:39.535861157 (497h3m39.406944202s) [upload=436256]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 08:32:39.535867620 (497h3m39.406950655s) [upload=436256]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 08:32:39.833497059 (497h3m39.704580164s) [upload=436256]: Done uploading segments...
Dec 26 08:32:39.833522858 (497h3m39.704605903s) [upload=436256]: Upload finished. err: <nil>
Duration: 49.451680519s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [...X........................................................] duration:
3.421930725s trace:
https://jaeger.dev.storj.io/trace/185fddc4dc40eba9
FAIL (2025-12-26 08:50:41 UTC)
orbiter warning: context deadline exceeded
Duration: 3.421999755s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [....X.......................................................] duration:
5.809803189s trace:
https://jaeger.dev.storj.io/trace/4b8b74dd4906ed4
FAIL (2025-12-26 08:50:28 UTC)
orbiter warning: context deadline exceeded
Duration: 5.809828498s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
21 out of 60 is failed (threshold=3) [X..X.XX.X.X..X..X.XXX.....X......X.X...X.XX..X.XXX..........] duration:
5.77323115s trace:
https://jaeger.dev.storj.io/trace/3312cc3668ef2c5f
FAIL (2025-12-26 08:50:02 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 26 08:42:18.506798665 (497h13m18.37788171s) [upload=436400]: Starting upload
Dec 26 08:42:18.506881102 (497h13m18.377964137s) [upload=436400]: Done starting upload
Dec 26 08:42:18.506963669 (497h13m18.378046714s) [upload=436400]: Uploading segments...
Dec 26 08:42:18.507170413 (497h13m18.378253478s) [upload=436400]: Got next segment. Inline: false
Dec 26 08:42:18.507247920 (497h13m18.378330965s) [upload=436400]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 26 08:42:18.507255454 (497h13m18.378338499s) [upload=436400]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 26 08:42:18.754283459 (497h13m18.625366504s) [seg_pos={0 0}, upload=436400]: Begin upload segment...
Dec 26 08:42:18.754356328 (497h13m18.625439373s) [seg_pos={0 0}, upload=436400]: Done begin upload segment.
Dec 26 08:42:18.754391745 (497h13m18.62547479s) [noise=true , address=145.79.227.40:4051 , piece_id=NLPGNO63SJLYA65HYN2CJBJFS6OTLX7ISULHSM6Q54PWOOTVAVWQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.754466177 (497h13m18.625549222s) [noise=true , address=77.90.31.17:4773 , piece_id=P645R4U4WSCWJR5XZSGFGVVF3QHAVSP6SN46CPQGBT6JWJB57ZIQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.755567554 (497h13m18.626650599s) [noise=true , address=77.90.31.18:4808 , piece_id=INW543LSVQLUQQ4FZZXVTVPV2LSGPKMUVHCSKE3OQ2CQAIEQFD2Q, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.755910858 (497h13m18.626993893s) [noise=true , address=145.79.227.17:4970 , piece_id=JXWYFTTKYNB6BSPBKWBK4YPF2PNAGWJKXHJAOUFIJVVIKIVNPUJA, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.756229915 (497h13m18.62731296s) [noise=true , address=145.79.227.46:4603 , piece_id=7L43455SH4JVSXBJYL3WVMRCWFQZ2CEI6DQA7MUXDYSKPPKESY7A, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.756613886 (497h13m18.627696922s) [noise=true , address=145.79.227.16:4912 , piece_id=TIPMXPQDX4HVH2NUQ3FUFT6J34SNM5DT2NHDXCZO76MMGPEEEHMQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.757020902 (497h13m18.628103937s) [noise=true , address=145.79.227.38:4977 , piece_id=MYMT3XBOXWU5DZOTPVHSVVXOKNHW7WWVCBECIO37VAYUTFK7XZSQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.757428849 (497h13m18.628511904s) [noise=true , address=77.90.31.11:4555 , piece_id=ZTSV63NMIHDFU464BAUD3MH3645MJOCGLTDQEKIRPD47EMLLO5EA, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.757902380 (497h13m18.628985425s) [noise=true , address=145.79.227.45:4229 , piece_id=GNXPJKQLV6ITICMBLE4PY2ODOKEJ6WWBUAPA74AO4IBVOVVEFSLA, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.757979276 (497h13m18.629062321s) [noise=true , address=77.90.31.21:4878 , piece_id=RNZ3EK2ZERZY6IQEBAZPHEUDBHNF7BARJF5FVZMLVVMLH5OK7WCQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.758365802 (497h13m18.629448837s) [noise=true , address=45.91.116.4:4937 , piece_id=7HT3WA63RZNVO3AX775OL37N4BJRXZCZ5P6A2A2Z6NPZX6RASSSQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.758773228 (497h13m18.629856263s) [noise=true , address=77.90.31.10:4517 , piece_id=RD3ARVUTAOQRS3YUQ7WL2LF47CXQFSVDDJSJUAEZAFVGUU4HJ7KQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.759342462 (497h13m18.630425507s) [noise=true , address=145.79.227.37:4567 , piece_id=J7Z3543ZQ2TIGFFHBO7V7GJKTGOY54Z6WAPYDI7ALYRYDS6NEK2Q, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.760026635 (497h13m18.63110968s) [noise=true , address=145.79.227.48:4301 , piece_id=IS2W2WAK7NOMKFEHY3QK7KS35XPQYC2723Q2ZJ27HTX4FPBQYHDQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.760798124 (497h13m18.631881169s) [noise=true , address=145.79.227.51:4411 , piece_id=7JDMY4F74SKL6NSFN2HO36EGUCM5MWVWR5HLXACHBLZXTDRRWI3Q, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.761677319 (497h13m18.632760364s) [noise=true , address=77.90.31.14:4664 , piece_id=JFEBDNYO2OQRCKCLBKWS43SZCZ7L2LSIF5FP2JS37FFARUEHCESA, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:20.466851608 (497h13m20.337934643s) [upload=436400]: Next returned nil segment
Dec 26 08:42:20.466865273 (497h13m20.337948318s) [upload=436400]: Waiting for error group managing segments...
Dec 26 08:42:20.750402218 (497h13m20.621485253s) [noise=true , address=145.79.227.37:4567 , piece_id=J7Z3543ZQ2TIGFFHBO7V7GJKTGOY54Z6WAPYDI7ALYRYDS6NEK2Q, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:20.763579740 (497h13m20.634662785s) [noise=true , address=145.79.227.48:4301 , piece_id=IS2W2WAK7NOMKFEHY3QK7KS35XPQYC2723Q2ZJ27HTX4FPBQYHDQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:20.767844120 (497h13m20.638927155s) [noise=true , address=145.79.227.38:4977 , piece_id=MYMT3XBOXWU5DZOTPVHSVVXOKNHW7WWVCBECIO37VAYUTFK7XZSQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:20.771941653 (497h13m20.643024698s) [noise=true , address=145.79.227.16:4912 , piece_id=TIPMXPQDX4HVH2NUQ3FUFT6J34SNM5DT2NHDXCZO76MMGPEEEHMQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:20.773887688 (497h13m20.644970723s) [noise=true , address=145.79.227.17:4970 , piece_id=JXWYFTTKYNB6BSPBKWBK4YPF2PNAGWJKXHJAOUFIJVVIKIVNPUJA, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:20.781414641 (497h13m20.652497676s) [noise=true , address=145.79.227.51:4411 , piece_id=7JDMY4F74SKL6NSFN2HO36EGUCM5MWVWR5HLXACHBLZXTDRRWI3Q, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:20.781591407 (497h13m20.652674452s) [noise=true , address=145.79.227.46:4603 , piece_id=7L43455SH4JVSXBJYL3WVMRCWFQZ2CEI6DQA7MUXDYSKPPKESY7A, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:20.787415879 (497h13m20.658498924s) [noise=true , address=145.79.227.45:4229 , piece_id=GNXPJKQLV6ITICMBLE4PY2ODOKEJ6WWBUAPA74AO4IBVOVVEFSLA, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:21.029225186 (497h13m20.900308221s) [noise=true , address=45.91.116.4:4937 , piece_id=7HT3WA63RZNVO3AX775OL37N4BJRXZCZ5P6A2A2Z6NPZX6RASSSQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:21.124342636 (497h13m20.995425671s) [noise=true , address=77.90.31.21:4878 , piece_id=RNZ3EK2ZERZY6IQEBAZPHEUDBHNF7BARJF5FVZMLVVMLH5OK7WCQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:21.148939092 (497h13m21.020022127s) [noise=true , address=77.90.31.14:4664 , piece_id=JFEBDNYO2OQRCKCLBKWS43SZCZ7L2LSIF5FP2JS37FFARUEHCESA, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:21.733228030 (497h13m21.604311075s) [noise=true , address=77.90.31.11:4555 , piece_id=ZTSV63NMIHDFU464BAUD3MH3645MJOCGLTDQEKIRPD47EMLLO5EA, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:21.818455067 (497h13m21.689538102s) [noise=true , address=77.90.31.17:4773 , piece_id=P645R4U4WSCWJR5XZSGFGVVF3QHAVSP6SN46CPQGBT6JWJB57ZIQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:21.818474194 (497h13m21.689557239s) [seg_pos={0 0}, upload=436400]: Segment reached optimal threshold of 13 pieces.
Dec 26 08:42:21.818682150 (497h13m21.689765195s) [noise=true , address=77.90.31.10:4517 , piece_id=RD3ARVUTAOQRS3YUQ7WL2LF47CXQFSVDDJSJUAEZAFVGUU4HJ7KQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF
Dec 26 08:42:21.818690926 (497h13m21.689773971s) [noise=true , address=77.90.31.10:4517 , piece_id=RD3ARVUTAOQRS3YUQ7WL2LF47CXQFSVDDJSJUAEZAFVGUU4HJ7KQ, seg_pos={0 0}, upload=436400]: Long tail cancellation due to optimal threshold
Dec 26 08:42:21.818911596 (497h13m21.689994631s) [noise=true , address=145.79.227.40:4051 , piece_id=NLPGNO63SJLYA65HYN2CJBJFS6OTLX7ISULHSM6Q54PWOOTVAVWQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF
Dec 26 08:42:21.818918088 (497h13m21.690001123s) [noise=true , address=145.79.227.40:4051 , piece_id=NLPGNO63SJLYA65HYN2CJBJFS6OTLX7ISULHSM6Q54PWOOTVAVWQ, seg_pos={0 0}, upload=436400]: Long tail cancellation due to optimal threshold
Dec 26 08:42:21.818941363 (497h13m21.690024418s) [noise=true , address=77.90.31.18:4808 , piece_id=INW543LSVQLUQQ4FZZXVTVPV2LSGPKMUVHCSKE3OQ2CQAIEQFD2Q, seg_pos={0 0}, upload=436400]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: expected piece hash; context canceled; EOF
Dec 26 08:42:21.818997390 (497h13m21.690080425s) [noise=true , address=77.90.31.18:4808 , piece_id=INW543LSVQLUQQ4FZZXVTVPV2LSGPKMUVHCSKE3OQ2CQAIEQFD2Q, seg_pos={0 0}, upload=436400]: Long tail cancellation due to optimal threshold
Dec 26 08:42:21.819032707 (497h13m21.690115742s) [seg_pos={0 0}, upload=436400]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 26 08:42:21.819224622 (497h13m21.690307657s) [upload=436400]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 08:42:21.819232357 (497h13m21.690315392s) [upload=436400]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 08:42:22.109308312 (497h13m21.980391357s) [upload=436400]: Done uploading segments...
Dec 26 08:42:22.109331417 (497h13m21.980414452s) [upload=436400]: Upload finished. err: <nil>
Duration: 5.773265886s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.....X......................................................] duration:
6.096296272s trace:
https://jaeger.dev.storj.io/trace/4b8b74dd4906ed4
FAIL (2025-12-26 08:50:02 UTC)
orbiter warning: context deadline exceeded
Duration: 6.096325579s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.......................................X....................] duration:
41.388880809s trace:
https://jaeger.dev.storj.io/trace/6820217d73ecf2a
FAIL (2025-12-26 08:49:53 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
ss=45.91.116.13:4174 , piece_id=RLKVQ3QJM4STACAOUTWA7STEIG4PITJRIB4YOI3GNTYMPA76IFHQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.705106031 (497h3m36.576189076s) [noise=true , address=77.90.31.19:4870 , piece_id=ZHJNUA7P36RBWXQXMDLFRIHQPSZNOPGL67AQVDTDEPCRY2R4EMWQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.705493759 (497h3m36.576576804s) [noise=true , address=145.79.227.20:4443 , piece_id=HAFVT75GMRHFRR5E5VD4AOJFHGRUS56MQX3HNHI6FQYYMZT4J4FQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.705630279 (497h3m36.576713334s) [noise=true , address=77.90.31.15:4717 , piece_id=CVL5BLBIJ5DWSL7ZSEJ4I62RENOFPNWWNZCSE62CTMWGSS2XTE2Q, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.705923348 (497h3m36.577006393s) [noise=true , address=77.90.31.12:4607 , piece_id=TPFAMNN2BEYPSYRYOMDMSS77ZRB35HOJYWMJGTJEERRWGV5KQHHA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706055580 (497h3m36.577138615s) [noise=true , address=64.49.13.169:4705 , piece_id=NG53QDRTA5IAV5W64W6MSZVNCWDUDHBB3DP4BF6G7DTF5DAF7ECA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706362764 (497h3m36.577445819s) [noise=true , address=77.90.31.11:4581 , piece_id=NQOGQYJLH7W7CXFLBDW3XUN6Y6YVQEWAFBMLNKNHKTU5VCV4V3GQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706507651 (497h3m36.577590686s) [noise=true , address=145.79.227.44:4226 , piece_id=XFE2F2254TY5CMO7J5O6S3RDKHOHSLQCUGDNHB37VKOXPM63GJEA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706728992 (497h3m36.577812027s) [noise=true , address=45.91.116.4:4942 , piece_id=VDKHG53ATFYJDQYDBXKZ7SN2RNAAZBYGQVGAF34PCG4UYO7XMFGA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706869399 (497h3m36.577952444s) [noise=true , address=77.90.31.16:4750 , piece_id=HIZWD4OHAZ6UKMODPFHABQRQW7ESEZPVMSD45RI5HGXPLHDJLJHQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707077395 (497h3m36.57816044s) [noise=true , address=145.79.227.34:4909 , piece_id=TSBSOK42K3BBLLTUBV6UFWW33FK67FZHJZTKEQFAAQ7ITKY3RAEQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707229524 (497h3m36.578312559s) [noise=true , address=77.90.31.13:4655 , piece_id=B6QS3X24QQRPWA3O5ENXR32XJOJZBZZOVVDMSESJYKU3XBIPGDIQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707443251 (497h3m36.578526296s) [noise=true , address=145.79.227.5:4241 , piece_id=ANDNWFXV5YWOH55WDLOMJXSKZVN45XSIYT2I4QB52DNO2PT3HYKQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707633815 (497h3m36.57871685s) [noise=true , address=64.49.13.154:4930 , piece_id=IHQYIUPLXMAKL4S76Y6NAGDP2ZWFRQBOXXJJENZ6JRGBXWLKDCFQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707813797 (497h3m36.578896832s) [noise=true , address=145.79.227.13:4822 , piece_id=HSHVLD6IYQOG3KQYCZGENM4ZYH7XCKGRAZLEL6N2WELIBI2LNHSA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.708022374 (497h3m36.579105439s) [noise=true , address=145.79.227.30:4775 , piece_id=OG65SPMQI3XHNJHUHKSJ4NZKGAB764OYFE7D5CBJCP3YRAGYANQA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.708682712 (497h3m36.579765747s) [noise=true , address=77.90.31.17:4788 , piece_id=CIK5XZI3QMG7LFEOKBPZGNLDOORNP6IQV3QYSHLIY5XSF3LLXBBQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:38.013251430 (497h3m37.884334475s) [upload=436256]: Next returned nil segment
Dec 26 08:32:38.013277480 (497h3m37.884360525s) [upload=436256]: Waiting for error group managing segments...
Dec 26 08:32:38.345076384 (497h3m38.216159419s) [noise=true , address=64.49.13.154:4930 , piece_id=IHQYIUPLXMAKL4S76Y6NAGDP2ZWFRQBOXXJJENZ6JRGBXWLKDCFQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.593234781 (497h3m38.464317836s) [noise=true , address=145.79.227.20:4443 , piece_id=HAFVT75GMRHFRR5E5VD4AOJFHGRUS56MQX3HNHI6FQYYMZT4J4FQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.593901300 (497h3m38.464984335s) [noise=true , address=145.79.227.5:4241 , piece_id=ANDNWFXV5YWOH55WDLOMJXSKZVN45XSIYT2I4QB52DNO2PT3HYKQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.594508065 (497h3m38.4655911s) [noise=true , address=145.79.227.13:4822 , piece_id=HSHVLD6IYQOG3KQYCZGENM4ZYH7XCKGRAZLEL6N2WELIBI2LNHSA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.636070587 (497h3m38.507153653s) [noise=true , address=145.79.227.34:4909 , piece_id=TSBSOK42K3BBLLTUBV6UFWW33FK67FZHJZTKEQFAAQ7ITKY3RAEQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.750629848 (497h3m38.621712883s) [noise=true , address=77.90.31.11:4581 , piece_id=NQOGQYJLH7W7CXFLBDW3XUN6Y6YVQEWAFBMLNKNHKTU5VCV4V3GQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.814987737 (497h3m38.686070782s) [noise=true , address=45.91.116.4:4942 , piece_id=VDKHG53ATFYJDQYDBXKZ7SN2RNAAZBYGQVGAF34PCG4UYO7XMFGA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.847327759 (497h3m38.718410794s) [noise=true , address=77.90.31.15:4717 , piece_id=CVL5BLBIJ5DWSL7ZSEJ4I62RENOFPNWWNZCSE62CTMWGSS2XTE2Q, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.902089014 (497h3m38.773172049s) [noise=true , address=45.91.116.13:4174 , piece_id=RLKVQ3QJM4STACAOUTWA7STEIG4PITJRIB4YOI3GNTYMPA76IFHQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.912841608 (497h3m38.783924644s) [noise=true , address=77.90.31.12:4607 , piece_id=TPFAMNN2BEYPSYRYOMDMSS77ZRB35HOJYWMJGTJEERRWGV5KQHHA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.070851326 (497h3m38.941934371s) [noise=true , address=77.90.31.16:4750 , piece_id=HIZWD4OHAZ6UKMODPFHABQRQW7ESEZPVMSD45RI5HGXPLHDJLJHQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.469018898 (497h3m39.340101933s) [noise=true , address=145.79.227.30:4775 , piece_id=OG65SPMQI3XHNJHUHKSJ4NZKGAB764OYFE7D5CBJCP3YRAGYANQA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.477948602 (497h3m39.349031637s) [noise=true , address=145.79.227.44:4226 , piece_id=XFE2F2254TY5CMO7J5O6S3RDKHOHSLQCUGDNHB37VKOXPM63GJEA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.534878005 (497h3m39.40596105s) [noise=true , address=77.90.31.17:4788 , piece_id=CIK5XZI3QMG7LFEOKBPZGNLDOORNP6IQV3QYSHLIY5XSF3LLXBBQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.534902853 (497h3m39.405985898s) [seg_pos={0 0}, upload=436256]: Segment reached optimal threshold of 14 pieces.
Dec 26 08:32:39.535131698 (497h3m39.406214743s) [noise=true , address=77.90.31.19:4870 , piece_id=ZHJNUA7P36RBWXQXMDLFRIHQPSZNOPGL67AQVDTDEPCRY2R4EMWQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hNVXEVe8DW8F8meERGWkxAazEZKLKBeprNn6e5C94eEoAMBaR): protocol: expected piece hash; context canceled; EOF
Dec 26 08:32:39.535140075 (497h3m39.40622311s) [noise=true , address=77.90.31.19:4870 , piece_id=ZHJNUA7P36RBWXQXMDLFRIHQPSZNOPGL67AQVDTDEPCRY2R4EMWQ, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold
Dec 26 08:32:39.535367978 (497h3m39.406451013s) [noise=true , address=77.90.31.18:4838 , piece_id=V5LDUMURFU4H43E2A5MTLWBI6ZJHVPYMUILBDV5N7ERELPLTIBAA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1if7vf1cNTBwAhb6owi6tuGLYqSfa3feWhsvGSevwuwykhz2RY): protocol: expected piece hash; context canceled; EOF
Dec 26 08:32:39.535375542 (497h3m39.406458577s) [noise=true , address=77.90.31.18:4838 , piece_id=V5LDUMURFU4H43E2A5MTLWBI6ZJHVPYMUILBDV5N7ERELPLTIBAA, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold
Dec 26 08:32:39.535565575 (497h3m39.40664861s) [noise=true , address=77.90.31.13:4655 , piece_id=B6QS3X24QQRPWA3O5ENXR32XJOJZBZZOVVDMSESJYKU3XBIPGDIQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1K7fQM1NVA1L7ukmDtyWh2DYWypUoj3GbreETZpPL6gXfXmZDD): protocol: expected piece hash; context canceled; EOF
Dec 26 08:32:39.535606041 (497h3m39.406689077s) [noise=true , address=77.90.31.13:4655 , piece_id=B6QS3X24QQRPWA3O5ENXR32XJOJZBZZOVVDMSESJYKU3XBIPGDIQ, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold
Dec 26 08:32:39.535707996 (497h3m39.406791031s) [noise=true , address=64.49.13.169:4705 , piece_id=NG53QDRTA5IAV5W64W6MSZVNCWDUDHBB3DP4BF6G7DTF5DAF7ECA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: read tcp 188.166.225.170:35726->64.49.13.169:4705: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 26 08:32:39.535715059 (497h3m39.406798094s) [noise=true , address=64.49.13.169:4705 , piece_id=NG53QDRTA5IAV5W64W6MSZVNCWDUDHBB3DP4BF6G7DTF5DAF7ECA, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold
Dec 26 08:32:39.535740878 (497h3m39.406823923s) [seg_pos={0 0}, upload=436256]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 26 08:32:39.535861157 (497h3m39.406944202s) [upload=436256]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 08:32:39.535867620 (497h3m39.406950655s) [upload=436256]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 08:32:39.833497059 (497h3m39.704580164s) [upload=436256]: Done uploading segments...
Dec 26 08:32:39.833522858 (497h3m39.704605903s) [upload=436256]: Upload finished. err: <nil>
Duration: 41.388900467s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [....X.......................................................] duration:
3.685320074s trace:
https://jaeger.dev.storj.io/trace/185fddc4dc40eba9
FAIL (2025-12-26 08:49:51 UTC)
orbiter warning: context deadline exceeded
Duration: 3.685367413s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [......X.....................................................] duration:
5.916594127s trace:
https://jaeger.dev.storj.io/trace/4b8b74dd4906ed4
FAIL (2025-12-26 08:49:47 UTC)
orbiter warning: context deadline exceeded
Duration: 5.916619385s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [.....X......................................................] duration:
3.466638575s trace:
https://jaeger.dev.storj.io/trace/185fddc4dc40eba9
FAIL (2025-12-26 08:49:35 UTC)
orbiter warning: context deadline exceeded
Duration: 3.466663032s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
21 out of 60 is failed (threshold=3) [.X..X.XX.X.X..X..X.XXX.....X......X.X...X.XX..X.XXX.........] duration:
5.759995187s trace:
https://jaeger.dev.storj.io/trace/3312cc3668ef2c5f
FAIL (2025-12-26 08:49:14 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 26 08:42:18.506798665 (497h13m18.37788171s) [upload=436400]: Starting upload
Dec 26 08:42:18.506881102 (497h13m18.377964137s) [upload=436400]: Done starting upload
Dec 26 08:42:18.506963669 (497h13m18.378046714s) [upload=436400]: Uploading segments...
Dec 26 08:42:18.507170413 (497h13m18.378253478s) [upload=436400]: Got next segment. Inline: false
Dec 26 08:42:18.507247920 (497h13m18.378330965s) [upload=436400]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 26 08:42:18.507255454 (497h13m18.378338499s) [upload=436400]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 26 08:42:18.754283459 (497h13m18.625366504s) [seg_pos={0 0}, upload=436400]: Begin upload segment...
Dec 26 08:42:18.754356328 (497h13m18.625439373s) [seg_pos={0 0}, upload=436400]: Done begin upload segment.
Dec 26 08:42:18.754391745 (497h13m18.62547479s) [noise=true , address=145.79.227.40:4051 , piece_id=NLPGNO63SJLYA65HYN2CJBJFS6OTLX7ISULHSM6Q54PWOOTVAVWQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.754466177 (497h13m18.625549222s) [noise=true , address=77.90.31.17:4773 , piece_id=P645R4U4WSCWJR5XZSGFGVVF3QHAVSP6SN46CPQGBT6JWJB57ZIQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.755567554 (497h13m18.626650599s) [noise=true , address=77.90.31.18:4808 , piece_id=INW543LSVQLUQQ4FZZXVTVPV2LSGPKMUVHCSKE3OQ2CQAIEQFD2Q, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.755910858 (497h13m18.626993893s) [noise=true , address=145.79.227.17:4970 , piece_id=JXWYFTTKYNB6BSPBKWBK4YPF2PNAGWJKXHJAOUFIJVVIKIVNPUJA, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.756229915 (497h13m18.62731296s) [noise=true , address=145.79.227.46:4603 , piece_id=7L43455SH4JVSXBJYL3WVMRCWFQZ2CEI6DQA7MUXDYSKPPKESY7A, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.756613886 (497h13m18.627696922s) [noise=true , address=145.79.227.16:4912 , piece_id=TIPMXPQDX4HVH2NUQ3FUFT6J34SNM5DT2NHDXCZO76MMGPEEEHMQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.757020902 (497h13m18.628103937s) [noise=true , address=145.79.227.38:4977 , piece_id=MYMT3XBOXWU5DZOTPVHSVVXOKNHW7WWVCBECIO37VAYUTFK7XZSQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.757428849 (497h13m18.628511904s) [noise=true , address=77.90.31.11:4555 , piece_id=ZTSV63NMIHDFU464BAUD3MH3645MJOCGLTDQEKIRPD47EMLLO5EA, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.757902380 (497h13m18.628985425s) [noise=true , address=145.79.227.45:4229 , piece_id=GNXPJKQLV6ITICMBLE4PY2ODOKEJ6WWBUAPA74AO4IBVOVVEFSLA, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.757979276 (497h13m18.629062321s) [noise=true , address=77.90.31.21:4878 , piece_id=RNZ3EK2ZERZY6IQEBAZPHEUDBHNF7BARJF5FVZMLVVMLH5OK7WCQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.758365802 (497h13m18.629448837s) [noise=true , address=45.91.116.4:4937 , piece_id=7HT3WA63RZNVO3AX775OL37N4BJRXZCZ5P6A2A2Z6NPZX6RASSSQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.758773228 (497h13m18.629856263s) [noise=true , address=77.90.31.10:4517 , piece_id=RD3ARVUTAOQRS3YUQ7WL2LF47CXQFSVDDJSJUAEZAFVGUU4HJ7KQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.759342462 (497h13m18.630425507s) [noise=true , address=145.79.227.37:4567 , piece_id=J7Z3543ZQ2TIGFFHBO7V7GJKTGOY54Z6WAPYDI7ALYRYDS6NEK2Q, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.760026635 (497h13m18.63110968s) [noise=true , address=145.79.227.48:4301 , piece_id=IS2W2WAK7NOMKFEHY3QK7KS35XPQYC2723Q2ZJ27HTX4FPBQYHDQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.760798124 (497h13m18.631881169s) [noise=true , address=145.79.227.51:4411 , piece_id=7JDMY4F74SKL6NSFN2HO36EGUCM5MWVWR5HLXACHBLZXTDRRWI3Q, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.761677319 (497h13m18.632760364s) [noise=true , address=77.90.31.14:4664 , piece_id=JFEBDNYO2OQRCKCLBKWS43SZCZ7L2LSIF5FP2JS37FFARUEHCESA, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:20.466851608 (497h13m20.337934643s) [upload=436400]: Next returned nil segment
Dec 26 08:42:20.466865273 (497h13m20.337948318s) [upload=436400]: Waiting for error group managing segments...
Dec 26 08:42:20.750402218 (497h13m20.621485253s) [noise=true , address=145.79.227.37:4567 , piece_id=J7Z3543ZQ2TIGFFHBO7V7GJKTGOY54Z6WAPYDI7ALYRYDS6NEK2Q, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:20.763579740 (497h13m20.634662785s) [noise=true , address=145.79.227.48:4301 , piece_id=IS2W2WAK7NOMKFEHY3QK7KS35XPQYC2723Q2ZJ27HTX4FPBQYHDQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:20.767844120 (497h13m20.638927155s) [noise=true , address=145.79.227.38:4977 , piece_id=MYMT3XBOXWU5DZOTPVHSVVXOKNHW7WWVCBECIO37VAYUTFK7XZSQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:20.771941653 (497h13m20.643024698s) [noise=true , address=145.79.227.16:4912 , piece_id=TIPMXPQDX4HVH2NUQ3FUFT6J34SNM5DT2NHDXCZO76MMGPEEEHMQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:20.773887688 (497h13m20.644970723s) [noise=true , address=145.79.227.17:4970 , piece_id=JXWYFTTKYNB6BSPBKWBK4YPF2PNAGWJKXHJAOUFIJVVIKIVNPUJA, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:20.781414641 (497h13m20.652497676s) [noise=true , address=145.79.227.51:4411 , piece_id=7JDMY4F74SKL6NSFN2HO36EGUCM5MWVWR5HLXACHBLZXTDRRWI3Q, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:20.781591407 (497h13m20.652674452s) [noise=true , address=145.79.227.46:4603 , piece_id=7L43455SH4JVSXBJYL3WVMRCWFQZ2CEI6DQA7MUXDYSKPPKESY7A, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:20.787415879 (497h13m20.658498924s) [noise=true , address=145.79.227.45:4229 , piece_id=GNXPJKQLV6ITICMBLE4PY2ODOKEJ6WWBUAPA74AO4IBVOVVEFSLA, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:21.029225186 (497h13m20.900308221s) [noise=true , address=45.91.116.4:4937 , piece_id=7HT3WA63RZNVO3AX775OL37N4BJRXZCZ5P6A2A2Z6NPZX6RASSSQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:21.124342636 (497h13m20.995425671s) [noise=true , address=77.90.31.21:4878 , piece_id=RNZ3EK2ZERZY6IQEBAZPHEUDBHNF7BARJF5FVZMLVVMLH5OK7WCQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:21.148939092 (497h13m21.020022127s) [noise=true , address=77.90.31.14:4664 , piece_id=JFEBDNYO2OQRCKCLBKWS43SZCZ7L2LSIF5FP2JS37FFARUEHCESA, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:21.733228030 (497h13m21.604311075s) [noise=true , address=77.90.31.11:4555 , piece_id=ZTSV63NMIHDFU464BAUD3MH3645MJOCGLTDQEKIRPD47EMLLO5EA, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:21.818455067 (497h13m21.689538102s) [noise=true , address=77.90.31.17:4773 , piece_id=P645R4U4WSCWJR5XZSGFGVVF3QHAVSP6SN46CPQGBT6JWJB57ZIQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:21.818474194 (497h13m21.689557239s) [seg_pos={0 0}, upload=436400]: Segment reached optimal threshold of 13 pieces.
Dec 26 08:42:21.818682150 (497h13m21.689765195s) [noise=true , address=77.90.31.10:4517 , piece_id=RD3ARVUTAOQRS3YUQ7WL2LF47CXQFSVDDJSJUAEZAFVGUU4HJ7KQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF
Dec 26 08:42:21.818690926 (497h13m21.689773971s) [noise=true , address=77.90.31.10:4517 , piece_id=RD3ARVUTAOQRS3YUQ7WL2LF47CXQFSVDDJSJUAEZAFVGUU4HJ7KQ, seg_pos={0 0}, upload=436400]: Long tail cancellation due to optimal threshold
Dec 26 08:42:21.818911596 (497h13m21.689994631s) [noise=true , address=145.79.227.40:4051 , piece_id=NLPGNO63SJLYA65HYN2CJBJFS6OTLX7ISULHSM6Q54PWOOTVAVWQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF
Dec 26 08:42:21.818918088 (497h13m21.690001123s) [noise=true , address=145.79.227.40:4051 , piece_id=NLPGNO63SJLYA65HYN2CJBJFS6OTLX7ISULHSM6Q54PWOOTVAVWQ, seg_pos={0 0}, upload=436400]: Long tail cancellation due to optimal threshold
Dec 26 08:42:21.818941363 (497h13m21.690024418s) [noise=true , address=77.90.31.18:4808 , piece_id=INW543LSVQLUQQ4FZZXVTVPV2LSGPKMUVHCSKE3OQ2CQAIEQFD2Q, seg_pos={0 0}, upload=436400]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: expected piece hash; context canceled; EOF
Dec 26 08:42:21.818997390 (497h13m21.690080425s) [noise=true , address=77.90.31.18:4808 , piece_id=INW543LSVQLUQQ4FZZXVTVPV2LSGPKMUVHCSKE3OQ2CQAIEQFD2Q, seg_pos={0 0}, upload=436400]: Long tail cancellation due to optimal threshold
Dec 26 08:42:21.819032707 (497h13m21.690115742s) [seg_pos={0 0}, upload=436400]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 26 08:42:21.819224622 (497h13m21.690307657s) [upload=436400]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 08:42:21.819232357 (497h13m21.690315392s) [upload=436400]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 08:42:22.109308312 (497h13m21.980391357s) [upload=436400]: Done uploading segments...
Dec 26 08:42:22.109331417 (497h13m21.980414452s) [upload=436400]: Upload finished. err: <nil>
Duration: 5.760013923s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [........................................X...................] duration:
31.688189698s trace:
https://jaeger.dev.storj.io/trace/6820217d73ecf2a
FAIL (2025-12-26 08:49:11 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
ss=45.91.116.13:4174 , piece_id=RLKVQ3QJM4STACAOUTWA7STEIG4PITJRIB4YOI3GNTYMPA76IFHQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.705106031 (497h3m36.576189076s) [noise=true , address=77.90.31.19:4870 , piece_id=ZHJNUA7P36RBWXQXMDLFRIHQPSZNOPGL67AQVDTDEPCRY2R4EMWQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.705493759 (497h3m36.576576804s) [noise=true , address=145.79.227.20:4443 , piece_id=HAFVT75GMRHFRR5E5VD4AOJFHGRUS56MQX3HNHI6FQYYMZT4J4FQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.705630279 (497h3m36.576713334s) [noise=true , address=77.90.31.15:4717 , piece_id=CVL5BLBIJ5DWSL7ZSEJ4I62RENOFPNWWNZCSE62CTMWGSS2XTE2Q, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.705923348 (497h3m36.577006393s) [noise=true , address=77.90.31.12:4607 , piece_id=TPFAMNN2BEYPSYRYOMDMSS77ZRB35HOJYWMJGTJEERRWGV5KQHHA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706055580 (497h3m36.577138615s) [noise=true , address=64.49.13.169:4705 , piece_id=NG53QDRTA5IAV5W64W6MSZVNCWDUDHBB3DP4BF6G7DTF5DAF7ECA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706362764 (497h3m36.577445819s) [noise=true , address=77.90.31.11:4581 , piece_id=NQOGQYJLH7W7CXFLBDW3XUN6Y6YVQEWAFBMLNKNHKTU5VCV4V3GQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706507651 (497h3m36.577590686s) [noise=true , address=145.79.227.44:4226 , piece_id=XFE2F2254TY5CMO7J5O6S3RDKHOHSLQCUGDNHB37VKOXPM63GJEA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706728992 (497h3m36.577812027s) [noise=true , address=45.91.116.4:4942 , piece_id=VDKHG53ATFYJDQYDBXKZ7SN2RNAAZBYGQVGAF34PCG4UYO7XMFGA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706869399 (497h3m36.577952444s) [noise=true , address=77.90.31.16:4750 , piece_id=HIZWD4OHAZ6UKMODPFHABQRQW7ESEZPVMSD45RI5HGXPLHDJLJHQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707077395 (497h3m36.57816044s) [noise=true , address=145.79.227.34:4909 , piece_id=TSBSOK42K3BBLLTUBV6UFWW33FK67FZHJZTKEQFAAQ7ITKY3RAEQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707229524 (497h3m36.578312559s) [noise=true , address=77.90.31.13:4655 , piece_id=B6QS3X24QQRPWA3O5ENXR32XJOJZBZZOVVDMSESJYKU3XBIPGDIQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707443251 (497h3m36.578526296s) [noise=true , address=145.79.227.5:4241 , piece_id=ANDNWFXV5YWOH55WDLOMJXSKZVN45XSIYT2I4QB52DNO2PT3HYKQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707633815 (497h3m36.57871685s) [noise=true , address=64.49.13.154:4930 , piece_id=IHQYIUPLXMAKL4S76Y6NAGDP2ZWFRQBOXXJJENZ6JRGBXWLKDCFQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707813797 (497h3m36.578896832s) [noise=true , address=145.79.227.13:4822 , piece_id=HSHVLD6IYQOG3KQYCZGENM4ZYH7XCKGRAZLEL6N2WELIBI2LNHSA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.708022374 (497h3m36.579105439s) [noise=true , address=145.79.227.30:4775 , piece_id=OG65SPMQI3XHNJHUHKSJ4NZKGAB764OYFE7D5CBJCP3YRAGYANQA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.708682712 (497h3m36.579765747s) [noise=true , address=77.90.31.17:4788 , piece_id=CIK5XZI3QMG7LFEOKBPZGNLDOORNP6IQV3QYSHLIY5XSF3LLXBBQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:38.013251430 (497h3m37.884334475s) [upload=436256]: Next returned nil segment
Dec 26 08:32:38.013277480 (497h3m37.884360525s) [upload=436256]: Waiting for error group managing segments...
Dec 26 08:32:38.345076384 (497h3m38.216159419s) [noise=true , address=64.49.13.154:4930 , piece_id=IHQYIUPLXMAKL4S76Y6NAGDP2ZWFRQBOXXJJENZ6JRGBXWLKDCFQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.593234781 (497h3m38.464317836s) [noise=true , address=145.79.227.20:4443 , piece_id=HAFVT75GMRHFRR5E5VD4AOJFHGRUS56MQX3HNHI6FQYYMZT4J4FQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.593901300 (497h3m38.464984335s) [noise=true , address=145.79.227.5:4241 , piece_id=ANDNWFXV5YWOH55WDLOMJXSKZVN45XSIYT2I4QB52DNO2PT3HYKQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.594508065 (497h3m38.4655911s) [noise=true , address=145.79.227.13:4822 , piece_id=HSHVLD6IYQOG3KQYCZGENM4ZYH7XCKGRAZLEL6N2WELIBI2LNHSA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.636070587 (497h3m38.507153653s) [noise=true , address=145.79.227.34:4909 , piece_id=TSBSOK42K3BBLLTUBV6UFWW33FK67FZHJZTKEQFAAQ7ITKY3RAEQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.750629848 (497h3m38.621712883s) [noise=true , address=77.90.31.11:4581 , piece_id=NQOGQYJLH7W7CXFLBDW3XUN6Y6YVQEWAFBMLNKNHKTU5VCV4V3GQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.814987737 (497h3m38.686070782s) [noise=true , address=45.91.116.4:4942 , piece_id=VDKHG53ATFYJDQYDBXKZ7SN2RNAAZBYGQVGAF34PCG4UYO7XMFGA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.847327759 (497h3m38.718410794s) [noise=true , address=77.90.31.15:4717 , piece_id=CVL5BLBIJ5DWSL7ZSEJ4I62RENOFPNWWNZCSE62CTMWGSS2XTE2Q, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.902089014 (497h3m38.773172049s) [noise=true , address=45.91.116.13:4174 , piece_id=RLKVQ3QJM4STACAOUTWA7STEIG4PITJRIB4YOI3GNTYMPA76IFHQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.912841608 (497h3m38.783924644s) [noise=true , address=77.90.31.12:4607 , piece_id=TPFAMNN2BEYPSYRYOMDMSS77ZRB35HOJYWMJGTJEERRWGV5KQHHA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.070851326 (497h3m38.941934371s) [noise=true , address=77.90.31.16:4750 , piece_id=HIZWD4OHAZ6UKMODPFHABQRQW7ESEZPVMSD45RI5HGXPLHDJLJHQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.469018898 (497h3m39.340101933s) [noise=true , address=145.79.227.30:4775 , piece_id=OG65SPMQI3XHNJHUHKSJ4NZKGAB764OYFE7D5CBJCP3YRAGYANQA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.477948602 (497h3m39.349031637s) [noise=true , address=145.79.227.44:4226 , piece_id=XFE2F2254TY5CMO7J5O6S3RDKHOHSLQCUGDNHB37VKOXPM63GJEA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.534878005 (497h3m39.40596105s) [noise=true , address=77.90.31.17:4788 , piece_id=CIK5XZI3QMG7LFEOKBPZGNLDOORNP6IQV3QYSHLIY5XSF3LLXBBQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.534902853 (497h3m39.405985898s) [seg_pos={0 0}, upload=436256]: Segment reached optimal threshold of 14 pieces.
Dec 26 08:32:39.535131698 (497h3m39.406214743s) [noise=true , address=77.90.31.19:4870 , piece_id=ZHJNUA7P36RBWXQXMDLFRIHQPSZNOPGL67AQVDTDEPCRY2R4EMWQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hNVXEVe8DW8F8meERGWkxAazEZKLKBeprNn6e5C94eEoAMBaR): protocol: expected piece hash; context canceled; EOF
Dec 26 08:32:39.535140075 (497h3m39.40622311s) [noise=true , address=77.90.31.19:4870 , piece_id=ZHJNUA7P36RBWXQXMDLFRIHQPSZNOPGL67AQVDTDEPCRY2R4EMWQ, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold
Dec 26 08:32:39.535367978 (497h3m39.406451013s) [noise=true , address=77.90.31.18:4838 , piece_id=V5LDUMURFU4H43E2A5MTLWBI6ZJHVPYMUILBDV5N7ERELPLTIBAA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1if7vf1cNTBwAhb6owi6tuGLYqSfa3feWhsvGSevwuwykhz2RY): protocol: expected piece hash; context canceled; EOF
Dec 26 08:32:39.535375542 (497h3m39.406458577s) [noise=true , address=77.90.31.18:4838 , piece_id=V5LDUMURFU4H43E2A5MTLWBI6ZJHVPYMUILBDV5N7ERELPLTIBAA, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold
Dec 26 08:32:39.535565575 (497h3m39.40664861s) [noise=true , address=77.90.31.13:4655 , piece_id=B6QS3X24QQRPWA3O5ENXR32XJOJZBZZOVVDMSESJYKU3XBIPGDIQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1K7fQM1NVA1L7ukmDtyWh2DYWypUoj3GbreETZpPL6gXfXmZDD): protocol: expected piece hash; context canceled; EOF
Dec 26 08:32:39.535606041 (497h3m39.406689077s) [noise=true , address=77.90.31.13:4655 , piece_id=B6QS3X24QQRPWA3O5ENXR32XJOJZBZZOVVDMSESJYKU3XBIPGDIQ, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold
Dec 26 08:32:39.535707996 (497h3m39.406791031s) [noise=true , address=64.49.13.169:4705 , piece_id=NG53QDRTA5IAV5W64W6MSZVNCWDUDHBB3DP4BF6G7DTF5DAF7ECA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: read tcp 188.166.225.170:35726->64.49.13.169:4705: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 26 08:32:39.535715059 (497h3m39.406798094s) [noise=true , address=64.49.13.169:4705 , piece_id=NG53QDRTA5IAV5W64W6MSZVNCWDUDHBB3DP4BF6G7DTF5DAF7ECA, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold
Dec 26 08:32:39.535740878 (497h3m39.406823923s) [seg_pos={0 0}, upload=436256]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 26 08:32:39.535861157 (497h3m39.406944202s) [upload=436256]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 08:32:39.535867620 (497h3m39.406950655s) [upload=436256]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 08:32:39.833497059 (497h3m39.704580164s) [upload=436256]: Done uploading segments...
Dec 26 08:32:39.833522858 (497h3m39.704605903s) [upload=436256]: Upload finished. err: <nil>
Duration: 31.688208595s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
21 out of 60 is failed (threshold=3) [..X..X.XX.X.X..X..X.XXX.....X......X.X...X.XX..X.XXX........] duration:
6.007045613s trace:
https://jaeger.dev.storj.io/trace/3312cc3668ef2c5f
FAIL (2025-12-26 08:48:41 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 26 08:42:18.506798665 (497h13m18.37788171s) [upload=436400]: Starting upload
Dec 26 08:42:18.506881102 (497h13m18.377964137s) [upload=436400]: Done starting upload
Dec 26 08:42:18.506963669 (497h13m18.378046714s) [upload=436400]: Uploading segments...
Dec 26 08:42:18.507170413 (497h13m18.378253478s) [upload=436400]: Got next segment. Inline: false
Dec 26 08:42:18.507247920 (497h13m18.378330965s) [upload=436400]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 26 08:42:18.507255454 (497h13m18.378338499s) [upload=436400]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 26 08:42:18.754283459 (497h13m18.625366504s) [seg_pos={0 0}, upload=436400]: Begin upload segment...
Dec 26 08:42:18.754356328 (497h13m18.625439373s) [seg_pos={0 0}, upload=436400]: Done begin upload segment.
Dec 26 08:42:18.754391745 (497h13m18.62547479s) [noise=true , address=145.79.227.40:4051 , piece_id=NLPGNO63SJLYA65HYN2CJBJFS6OTLX7ISULHSM6Q54PWOOTVAVWQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.754466177 (497h13m18.625549222s) [noise=true , address=77.90.31.17:4773 , piece_id=P645R4U4WSCWJR5XZSGFGVVF3QHAVSP6SN46CPQGBT6JWJB57ZIQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.755567554 (497h13m18.626650599s) [noise=true , address=77.90.31.18:4808 , piece_id=INW543LSVQLUQQ4FZZXVTVPV2LSGPKMUVHCSKE3OQ2CQAIEQFD2Q, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.755910858 (497h13m18.626993893s) [noise=true , address=145.79.227.17:4970 , piece_id=JXWYFTTKYNB6BSPBKWBK4YPF2PNAGWJKXHJAOUFIJVVIKIVNPUJA, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.756229915 (497h13m18.62731296s) [noise=true , address=145.79.227.46:4603 , piece_id=7L43455SH4JVSXBJYL3WVMRCWFQZ2CEI6DQA7MUXDYSKPPKESY7A, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.756613886 (497h13m18.627696922s) [noise=true , address=145.79.227.16:4912 , piece_id=TIPMXPQDX4HVH2NUQ3FUFT6J34SNM5DT2NHDXCZO76MMGPEEEHMQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.757020902 (497h13m18.628103937s) [noise=true , address=145.79.227.38:4977 , piece_id=MYMT3XBOXWU5DZOTPVHSVVXOKNHW7WWVCBECIO37VAYUTFK7XZSQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.757428849 (497h13m18.628511904s) [noise=true , address=77.90.31.11:4555 , piece_id=ZTSV63NMIHDFU464BAUD3MH3645MJOCGLTDQEKIRPD47EMLLO5EA, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.757902380 (497h13m18.628985425s) [noise=true , address=145.79.227.45:4229 , piece_id=GNXPJKQLV6ITICMBLE4PY2ODOKEJ6WWBUAPA74AO4IBVOVVEFSLA, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.757979276 (497h13m18.629062321s) [noise=true , address=77.90.31.21:4878 , piece_id=RNZ3EK2ZERZY6IQEBAZPHEUDBHNF7BARJF5FVZMLVVMLH5OK7WCQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.758365802 (497h13m18.629448837s) [noise=true , address=45.91.116.4:4937 , piece_id=7HT3WA63RZNVO3AX775OL37N4BJRXZCZ5P6A2A2Z6NPZX6RASSSQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.758773228 (497h13m18.629856263s) [noise=true , address=77.90.31.10:4517 , piece_id=RD3ARVUTAOQRS3YUQ7WL2LF47CXQFSVDDJSJUAEZAFVGUU4HJ7KQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.759342462 (497h13m18.630425507s) [noise=true , address=145.79.227.37:4567 , piece_id=J7Z3543ZQ2TIGFFHBO7V7GJKTGOY54Z6WAPYDI7ALYRYDS6NEK2Q, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.760026635 (497h13m18.63110968s) [noise=true , address=145.79.227.48:4301 , piece_id=IS2W2WAK7NOMKFEHY3QK7KS35XPQYC2723Q2ZJ27HTX4FPBQYHDQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.760798124 (497h13m18.631881169s) [noise=true , address=145.79.227.51:4411 , piece_id=7JDMY4F74SKL6NSFN2HO36EGUCM5MWVWR5HLXACHBLZXTDRRWI3Q, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.761677319 (497h13m18.632760364s) [noise=true , address=77.90.31.14:4664 , piece_id=JFEBDNYO2OQRCKCLBKWS43SZCZ7L2LSIF5FP2JS37FFARUEHCESA, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:20.466851608 (497h13m20.337934643s) [upload=436400]: Next returned nil segment
Dec 26 08:42:20.466865273 (497h13m20.337948318s) [upload=436400]: Waiting for error group managing segments...
Dec 26 08:42:20.750402218 (497h13m20.621485253s) [noise=true , address=145.79.227.37:4567 , piece_id=J7Z3543ZQ2TIGFFHBO7V7GJKTGOY54Z6WAPYDI7ALYRYDS6NEK2Q, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:20.763579740 (497h13m20.634662785s) [noise=true , address=145.79.227.48:4301 , piece_id=IS2W2WAK7NOMKFEHY3QK7KS35XPQYC2723Q2ZJ27HTX4FPBQYHDQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:20.767844120 (497h13m20.638927155s) [noise=true , address=145.79.227.38:4977 , piece_id=MYMT3XBOXWU5DZOTPVHSVVXOKNHW7WWVCBECIO37VAYUTFK7XZSQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:20.771941653 (497h13m20.643024698s) [noise=true , address=145.79.227.16:4912 , piece_id=TIPMXPQDX4HVH2NUQ3FUFT6J34SNM5DT2NHDXCZO76MMGPEEEHMQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:20.773887688 (497h13m20.644970723s) [noise=true , address=145.79.227.17:4970 , piece_id=JXWYFTTKYNB6BSPBKWBK4YPF2PNAGWJKXHJAOUFIJVVIKIVNPUJA, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:20.781414641 (497h13m20.652497676s) [noise=true , address=145.79.227.51:4411 , piece_id=7JDMY4F74SKL6NSFN2HO36EGUCM5MWVWR5HLXACHBLZXTDRRWI3Q, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:20.781591407 (497h13m20.652674452s) [noise=true , address=145.79.227.46:4603 , piece_id=7L43455SH4JVSXBJYL3WVMRCWFQZ2CEI6DQA7MUXDYSKPPKESY7A, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:20.787415879 (497h13m20.658498924s) [noise=true , address=145.79.227.45:4229 , piece_id=GNXPJKQLV6ITICMBLE4PY2ODOKEJ6WWBUAPA74AO4IBVOVVEFSLA, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:21.029225186 (497h13m20.900308221s) [noise=true , address=45.91.116.4:4937 , piece_id=7HT3WA63RZNVO3AX775OL37N4BJRXZCZ5P6A2A2Z6NPZX6RASSSQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:21.124342636 (497h13m20.995425671s) [noise=true , address=77.90.31.21:4878 , piece_id=RNZ3EK2ZERZY6IQEBAZPHEUDBHNF7BARJF5FVZMLVVMLH5OK7WCQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:21.148939092 (497h13m21.020022127s) [noise=true , address=77.90.31.14:4664 , piece_id=JFEBDNYO2OQRCKCLBKWS43SZCZ7L2LSIF5FP2JS37FFARUEHCESA, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:21.733228030 (497h13m21.604311075s) [noise=true , address=77.90.31.11:4555 , piece_id=ZTSV63NMIHDFU464BAUD3MH3645MJOCGLTDQEKIRPD47EMLLO5EA, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:21.818455067 (497h13m21.689538102s) [noise=true , address=77.90.31.17:4773 , piece_id=P645R4U4WSCWJR5XZSGFGVVF3QHAVSP6SN46CPQGBT6JWJB57ZIQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:21.818474194 (497h13m21.689557239s) [seg_pos={0 0}, upload=436400]: Segment reached optimal threshold of 13 pieces.
Dec 26 08:42:21.818682150 (497h13m21.689765195s) [noise=true , address=77.90.31.10:4517 , piece_id=RD3ARVUTAOQRS3YUQ7WL2LF47CXQFSVDDJSJUAEZAFVGUU4HJ7KQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF
Dec 26 08:42:21.818690926 (497h13m21.689773971s) [noise=true , address=77.90.31.10:4517 , piece_id=RD3ARVUTAOQRS3YUQ7WL2LF47CXQFSVDDJSJUAEZAFVGUU4HJ7KQ, seg_pos={0 0}, upload=436400]: Long tail cancellation due to optimal threshold
Dec 26 08:42:21.818911596 (497h13m21.689994631s) [noise=true , address=145.79.227.40:4051 , piece_id=NLPGNO63SJLYA65HYN2CJBJFS6OTLX7ISULHSM6Q54PWOOTVAVWQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF
Dec 26 08:42:21.818918088 (497h13m21.690001123s) [noise=true , address=145.79.227.40:4051 , piece_id=NLPGNO63SJLYA65HYN2CJBJFS6OTLX7ISULHSM6Q54PWOOTVAVWQ, seg_pos={0 0}, upload=436400]: Long tail cancellation due to optimal threshold
Dec 26 08:42:21.818941363 (497h13m21.690024418s) [noise=true , address=77.90.31.18:4808 , piece_id=INW543LSVQLUQQ4FZZXVTVPV2LSGPKMUVHCSKE3OQ2CQAIEQFD2Q, seg_pos={0 0}, upload=436400]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: expected piece hash; context canceled; EOF
Dec 26 08:42:21.818997390 (497h13m21.690080425s) [noise=true , address=77.90.31.18:4808 , piece_id=INW543LSVQLUQQ4FZZXVTVPV2LSGPKMUVHCSKE3OQ2CQAIEQFD2Q, seg_pos={0 0}, upload=436400]: Long tail cancellation due to optimal threshold
Dec 26 08:42:21.819032707 (497h13m21.690115742s) [seg_pos={0 0}, upload=436400]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 26 08:42:21.819224622 (497h13m21.690307657s) [upload=436400]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 08:42:21.819232357 (497h13m21.690315392s) [upload=436400]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 08:42:22.109308312 (497h13m21.980391357s) [upload=436400]: Done uploading segments...
Dec 26 08:42:22.109331417 (497h13m21.980414452s) [upload=436400]: Upload finished. err: <nil>
Duration: 6.007068666s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.........................................X..................] duration:
47.172908629s trace:
https://jaeger.dev.storj.io/trace/6820217d73ecf2a
FAIL (2025-12-26 08:48:39 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
ss=45.91.116.13:4174 , piece_id=RLKVQ3QJM4STACAOUTWA7STEIG4PITJRIB4YOI3GNTYMPA76IFHQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.705106031 (497h3m36.576189076s) [noise=true , address=77.90.31.19:4870 , piece_id=ZHJNUA7P36RBWXQXMDLFRIHQPSZNOPGL67AQVDTDEPCRY2R4EMWQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.705493759 (497h3m36.576576804s) [noise=true , address=145.79.227.20:4443 , piece_id=HAFVT75GMRHFRR5E5VD4AOJFHGRUS56MQX3HNHI6FQYYMZT4J4FQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.705630279 (497h3m36.576713334s) [noise=true , address=77.90.31.15:4717 , piece_id=CVL5BLBIJ5DWSL7ZSEJ4I62RENOFPNWWNZCSE62CTMWGSS2XTE2Q, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.705923348 (497h3m36.577006393s) [noise=true , address=77.90.31.12:4607 , piece_id=TPFAMNN2BEYPSYRYOMDMSS77ZRB35HOJYWMJGTJEERRWGV5KQHHA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706055580 (497h3m36.577138615s) [noise=true , address=64.49.13.169:4705 , piece_id=NG53QDRTA5IAV5W64W6MSZVNCWDUDHBB3DP4BF6G7DTF5DAF7ECA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706362764 (497h3m36.577445819s) [noise=true , address=77.90.31.11:4581 , piece_id=NQOGQYJLH7W7CXFLBDW3XUN6Y6YVQEWAFBMLNKNHKTU5VCV4V3GQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706507651 (497h3m36.577590686s) [noise=true , address=145.79.227.44:4226 , piece_id=XFE2F2254TY5CMO7J5O6S3RDKHOHSLQCUGDNHB37VKOXPM63GJEA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706728992 (497h3m36.577812027s) [noise=true , address=45.91.116.4:4942 , piece_id=VDKHG53ATFYJDQYDBXKZ7SN2RNAAZBYGQVGAF34PCG4UYO7XMFGA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706869399 (497h3m36.577952444s) [noise=true , address=77.90.31.16:4750 , piece_id=HIZWD4OHAZ6UKMODPFHABQRQW7ESEZPVMSD45RI5HGXPLHDJLJHQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707077395 (497h3m36.57816044s) [noise=true , address=145.79.227.34:4909 , piece_id=TSBSOK42K3BBLLTUBV6UFWW33FK67FZHJZTKEQFAAQ7ITKY3RAEQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707229524 (497h3m36.578312559s) [noise=true , address=77.90.31.13:4655 , piece_id=B6QS3X24QQRPWA3O5ENXR32XJOJZBZZOVVDMSESJYKU3XBIPGDIQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707443251 (497h3m36.578526296s) [noise=true , address=145.79.227.5:4241 , piece_id=ANDNWFXV5YWOH55WDLOMJXSKZVN45XSIYT2I4QB52DNO2PT3HYKQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707633815 (497h3m36.57871685s) [noise=true , address=64.49.13.154:4930 , piece_id=IHQYIUPLXMAKL4S76Y6NAGDP2ZWFRQBOXXJJENZ6JRGBXWLKDCFQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707813797 (497h3m36.578896832s) [noise=true , address=145.79.227.13:4822 , piece_id=HSHVLD6IYQOG3KQYCZGENM4ZYH7XCKGRAZLEL6N2WELIBI2LNHSA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.708022374 (497h3m36.579105439s) [noise=true , address=145.79.227.30:4775 , piece_id=OG65SPMQI3XHNJHUHKSJ4NZKGAB764OYFE7D5CBJCP3YRAGYANQA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.708682712 (497h3m36.579765747s) [noise=true , address=77.90.31.17:4788 , piece_id=CIK5XZI3QMG7LFEOKBPZGNLDOORNP6IQV3QYSHLIY5XSF3LLXBBQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:38.013251430 (497h3m37.884334475s) [upload=436256]: Next returned nil segment
Dec 26 08:32:38.013277480 (497h3m37.884360525s) [upload=436256]: Waiting for error group managing segments...
Dec 26 08:32:38.345076384 (497h3m38.216159419s) [noise=true , address=64.49.13.154:4930 , piece_id=IHQYIUPLXMAKL4S76Y6NAGDP2ZWFRQBOXXJJENZ6JRGBXWLKDCFQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.593234781 (497h3m38.464317836s) [noise=true , address=145.79.227.20:4443 , piece_id=HAFVT75GMRHFRR5E5VD4AOJFHGRUS56MQX3HNHI6FQYYMZT4J4FQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.593901300 (497h3m38.464984335s) [noise=true , address=145.79.227.5:4241 , piece_id=ANDNWFXV5YWOH55WDLOMJXSKZVN45XSIYT2I4QB52DNO2PT3HYKQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.594508065 (497h3m38.4655911s) [noise=true , address=145.79.227.13:4822 , piece_id=HSHVLD6IYQOG3KQYCZGENM4ZYH7XCKGRAZLEL6N2WELIBI2LNHSA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.636070587 (497h3m38.507153653s) [noise=true , address=145.79.227.34:4909 , piece_id=TSBSOK42K3BBLLTUBV6UFWW33FK67FZHJZTKEQFAAQ7ITKY3RAEQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.750629848 (497h3m38.621712883s) [noise=true , address=77.90.31.11:4581 , piece_id=NQOGQYJLH7W7CXFLBDW3XUN6Y6YVQEWAFBMLNKNHKTU5VCV4V3GQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.814987737 (497h3m38.686070782s) [noise=true , address=45.91.116.4:4942 , piece_id=VDKHG53ATFYJDQYDBXKZ7SN2RNAAZBYGQVGAF34PCG4UYO7XMFGA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.847327759 (497h3m38.718410794s) [noise=true , address=77.90.31.15:4717 , piece_id=CVL5BLBIJ5DWSL7ZSEJ4I62RENOFPNWWNZCSE62CTMWGSS2XTE2Q, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.902089014 (497h3m38.773172049s) [noise=true , address=45.91.116.13:4174 , piece_id=RLKVQ3QJM4STACAOUTWA7STEIG4PITJRIB4YOI3GNTYMPA76IFHQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.912841608 (497h3m38.783924644s) [noise=true , address=77.90.31.12:4607 , piece_id=TPFAMNN2BEYPSYRYOMDMSS77ZRB35HOJYWMJGTJEERRWGV5KQHHA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.070851326 (497h3m38.941934371s) [noise=true , address=77.90.31.16:4750 , piece_id=HIZWD4OHAZ6UKMODPFHABQRQW7ESEZPVMSD45RI5HGXPLHDJLJHQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.469018898 (497h3m39.340101933s) [noise=true , address=145.79.227.30:4775 , piece_id=OG65SPMQI3XHNJHUHKSJ4NZKGAB764OYFE7D5CBJCP3YRAGYANQA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.477948602 (497h3m39.349031637s) [noise=true , address=145.79.227.44:4226 , piece_id=XFE2F2254TY5CMO7J5O6S3RDKHOHSLQCUGDNHB37VKOXPM63GJEA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.534878005 (497h3m39.40596105s) [noise=true , address=77.90.31.17:4788 , piece_id=CIK5XZI3QMG7LFEOKBPZGNLDOORNP6IQV3QYSHLIY5XSF3LLXBBQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.534902853 (497h3m39.405985898s) [seg_pos={0 0}, upload=436256]: Segment reached optimal threshold of 14 pieces.
Dec 26 08:32:39.535131698 (497h3m39.406214743s) [noise=true , address=77.90.31.19:4870 , piece_id=ZHJNUA7P36RBWXQXMDLFRIHQPSZNOPGL67AQVDTDEPCRY2R4EMWQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hNVXEVe8DW8F8meERGWkxAazEZKLKBeprNn6e5C94eEoAMBaR): protocol: expected piece hash; context canceled; EOF
Dec 26 08:32:39.535140075 (497h3m39.40622311s) [noise=true , address=77.90.31.19:4870 , piece_id=ZHJNUA7P36RBWXQXMDLFRIHQPSZNOPGL67AQVDTDEPCRY2R4EMWQ, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold
Dec 26 08:32:39.535367978 (497h3m39.406451013s) [noise=true , address=77.90.31.18:4838 , piece_id=V5LDUMURFU4H43E2A5MTLWBI6ZJHVPYMUILBDV5N7ERELPLTIBAA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1if7vf1cNTBwAhb6owi6tuGLYqSfa3feWhsvGSevwuwykhz2RY): protocol: expected piece hash; context canceled; EOF
Dec 26 08:32:39.535375542 (497h3m39.406458577s) [noise=true , address=77.90.31.18:4838 , piece_id=V5LDUMURFU4H43E2A5MTLWBI6ZJHVPYMUILBDV5N7ERELPLTIBAA, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold
Dec 26 08:32:39.535565575 (497h3m39.40664861s) [noise=true , address=77.90.31.13:4655 , piece_id=B6QS3X24QQRPWA3O5ENXR32XJOJZBZZOVVDMSESJYKU3XBIPGDIQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1K7fQM1NVA1L7ukmDtyWh2DYWypUoj3GbreETZpPL6gXfXmZDD): protocol: expected piece hash; context canceled; EOF
Dec 26 08:32:39.535606041 (497h3m39.406689077s) [noise=true , address=77.90.31.13:4655 , piece_id=B6QS3X24QQRPWA3O5ENXR32XJOJZBZZOVVDMSESJYKU3XBIPGDIQ, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold
Dec 26 08:32:39.535707996 (497h3m39.406791031s) [noise=true , address=64.49.13.169:4705 , piece_id=NG53QDRTA5IAV5W64W6MSZVNCWDUDHBB3DP4BF6G7DTF5DAF7ECA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: read tcp 188.166.225.170:35726->64.49.13.169:4705: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 26 08:32:39.535715059 (497h3m39.406798094s) [noise=true , address=64.49.13.169:4705 , piece_id=NG53QDRTA5IAV5W64W6MSZVNCWDUDHBB3DP4BF6G7DTF5DAF7ECA, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold
Dec 26 08:32:39.535740878 (497h3m39.406823923s) [seg_pos={0 0}, upload=436256]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 26 08:32:39.535861157 (497h3m39.406944202s) [upload=436256]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 08:32:39.535867620 (497h3m39.406950655s) [upload=436256]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 08:32:39.833497059 (497h3m39.704580164s) [upload=436256]: Done uploading segments...
Dec 26 08:32:39.833522858 (497h3m39.704605903s) [upload=436256]: Upload finished. err: <nil>
Duration: 47.17295125s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.......X....................................................] duration:
5.945844896s trace:
https://jaeger.dev.storj.io/trace/4b8b74dd4906ed4
FAIL (2025-12-26 08:48:38 UTC)
orbiter warning: context deadline exceeded
Duration: 5.945869753s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [......X.....................................................] duration:
3.43122412s trace:
https://jaeger.dev.storj.io/trace/185fddc4dc40eba9
FAIL (2025-12-26 08:48:25 UTC)
orbiter warning: context deadline exceeded
Duration: 3.431253505s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
21 out of 60 is failed (threshold=3) [...X..X.XX.X.X..X..X.XXX.....X......X.X...X.XX..X.XXX.......] duration:
5.923175712s trace:
https://jaeger.dev.storj.io/trace/3312cc3668ef2c5f
FAIL (2025-12-26 08:48:00 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 26 08:42:18.506798665 (497h13m18.37788171s) [upload=436400]: Starting upload
Dec 26 08:42:18.506881102 (497h13m18.377964137s) [upload=436400]: Done starting upload
Dec 26 08:42:18.506963669 (497h13m18.378046714s) [upload=436400]: Uploading segments...
Dec 26 08:42:18.507170413 (497h13m18.378253478s) [upload=436400]: Got next segment. Inline: false
Dec 26 08:42:18.507247920 (497h13m18.378330965s) [upload=436400]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 26 08:42:18.507255454 (497h13m18.378338499s) [upload=436400]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 26 08:42:18.754283459 (497h13m18.625366504s) [seg_pos={0 0}, upload=436400]: Begin upload segment...
Dec 26 08:42:18.754356328 (497h13m18.625439373s) [seg_pos={0 0}, upload=436400]: Done begin upload segment.
Dec 26 08:42:18.754391745 (497h13m18.62547479s) [noise=true , address=145.79.227.40:4051 , piece_id=NLPGNO63SJLYA65HYN2CJBJFS6OTLX7ISULHSM6Q54PWOOTVAVWQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.754466177 (497h13m18.625549222s) [noise=true , address=77.90.31.17:4773 , piece_id=P645R4U4WSCWJR5XZSGFGVVF3QHAVSP6SN46CPQGBT6JWJB57ZIQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.755567554 (497h13m18.626650599s) [noise=true , address=77.90.31.18:4808 , piece_id=INW543LSVQLUQQ4FZZXVTVPV2LSGPKMUVHCSKE3OQ2CQAIEQFD2Q, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.755910858 (497h13m18.626993893s) [noise=true , address=145.79.227.17:4970 , piece_id=JXWYFTTKYNB6BSPBKWBK4YPF2PNAGWJKXHJAOUFIJVVIKIVNPUJA, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.756229915 (497h13m18.62731296s) [noise=true , address=145.79.227.46:4603 , piece_id=7L43455SH4JVSXBJYL3WVMRCWFQZ2CEI6DQA7MUXDYSKPPKESY7A, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.756613886 (497h13m18.627696922s) [noise=true , address=145.79.227.16:4912 , piece_id=TIPMXPQDX4HVH2NUQ3FUFT6J34SNM5DT2NHDXCZO76MMGPEEEHMQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.757020902 (497h13m18.628103937s) [noise=true , address=145.79.227.38:4977 , piece_id=MYMT3XBOXWU5DZOTPVHSVVXOKNHW7WWVCBECIO37VAYUTFK7XZSQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.757428849 (497h13m18.628511904s) [noise=true , address=77.90.31.11:4555 , piece_id=ZTSV63NMIHDFU464BAUD3MH3645MJOCGLTDQEKIRPD47EMLLO5EA, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.757902380 (497h13m18.628985425s) [noise=true , address=145.79.227.45:4229 , piece_id=GNXPJKQLV6ITICMBLE4PY2ODOKEJ6WWBUAPA74AO4IBVOVVEFSLA, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.757979276 (497h13m18.629062321s) [noise=true , address=77.90.31.21:4878 , piece_id=RNZ3EK2ZERZY6IQEBAZPHEUDBHNF7BARJF5FVZMLVVMLH5OK7WCQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.758365802 (497h13m18.629448837s) [noise=true , address=45.91.116.4:4937 , piece_id=7HT3WA63RZNVO3AX775OL37N4BJRXZCZ5P6A2A2Z6NPZX6RASSSQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.758773228 (497h13m18.629856263s) [noise=true , address=77.90.31.10:4517 , piece_id=RD3ARVUTAOQRS3YUQ7WL2LF47CXQFSVDDJSJUAEZAFVGUU4HJ7KQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.759342462 (497h13m18.630425507s) [noise=true , address=145.79.227.37:4567 , piece_id=J7Z3543ZQ2TIGFFHBO7V7GJKTGOY54Z6WAPYDI7ALYRYDS6NEK2Q, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.760026635 (497h13m18.63110968s) [noise=true , address=145.79.227.48:4301 , piece_id=IS2W2WAK7NOMKFEHY3QK7KS35XPQYC2723Q2ZJ27HTX4FPBQYHDQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.760798124 (497h13m18.631881169s) [noise=true , address=145.79.227.51:4411 , piece_id=7JDMY4F74SKL6NSFN2HO36EGUCM5MWVWR5HLXACHBLZXTDRRWI3Q, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.761677319 (497h13m18.632760364s) [noise=true , address=77.90.31.14:4664 , piece_id=JFEBDNYO2OQRCKCLBKWS43SZCZ7L2LSIF5FP2JS37FFARUEHCESA, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:20.466851608 (497h13m20.337934643s) [upload=436400]: Next returned nil segment
Dec 26 08:42:20.466865273 (497h13m20.337948318s) [upload=436400]: Waiting for error group managing segments...
Dec 26 08:42:20.750402218 (497h13m20.621485253s) [noise=true , address=145.79.227.37:4567 , piece_id=J7Z3543ZQ2TIGFFHBO7V7GJKTGOY54Z6WAPYDI7ALYRYDS6NEK2Q, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:20.763579740 (497h13m20.634662785s) [noise=true , address=145.79.227.48:4301 , piece_id=IS2W2WAK7NOMKFEHY3QK7KS35XPQYC2723Q2ZJ27HTX4FPBQYHDQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:20.767844120 (497h13m20.638927155s) [noise=true , address=145.79.227.38:4977 , piece_id=MYMT3XBOXWU5DZOTPVHSVVXOKNHW7WWVCBECIO37VAYUTFK7XZSQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:20.771941653 (497h13m20.643024698s) [noise=true , address=145.79.227.16:4912 , piece_id=TIPMXPQDX4HVH2NUQ3FUFT6J34SNM5DT2NHDXCZO76MMGPEEEHMQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:20.773887688 (497h13m20.644970723s) [noise=true , address=145.79.227.17:4970 , piece_id=JXWYFTTKYNB6BSPBKWBK4YPF2PNAGWJKXHJAOUFIJVVIKIVNPUJA, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:20.781414641 (497h13m20.652497676s) [noise=true , address=145.79.227.51:4411 , piece_id=7JDMY4F74SKL6NSFN2HO36EGUCM5MWVWR5HLXACHBLZXTDRRWI3Q, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:20.781591407 (497h13m20.652674452s) [noise=true , address=145.79.227.46:4603 , piece_id=7L43455SH4JVSXBJYL3WVMRCWFQZ2CEI6DQA7MUXDYSKPPKESY7A, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:20.787415879 (497h13m20.658498924s) [noise=true , address=145.79.227.45:4229 , piece_id=GNXPJKQLV6ITICMBLE4PY2ODOKEJ6WWBUAPA74AO4IBVOVVEFSLA, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:21.029225186 (497h13m20.900308221s) [noise=true , address=45.91.116.4:4937 , piece_id=7HT3WA63RZNVO3AX775OL37N4BJRXZCZ5P6A2A2Z6NPZX6RASSSQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:21.124342636 (497h13m20.995425671s) [noise=true , address=77.90.31.21:4878 , piece_id=RNZ3EK2ZERZY6IQEBAZPHEUDBHNF7BARJF5FVZMLVVMLH5OK7WCQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:21.148939092 (497h13m21.020022127s) [noise=true , address=77.90.31.14:4664 , piece_id=JFEBDNYO2OQRCKCLBKWS43SZCZ7L2LSIF5FP2JS37FFARUEHCESA, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:21.733228030 (497h13m21.604311075s) [noise=true , address=77.90.31.11:4555 , piece_id=ZTSV63NMIHDFU464BAUD3MH3645MJOCGLTDQEKIRPD47EMLLO5EA, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:21.818455067 (497h13m21.689538102s) [noise=true , address=77.90.31.17:4773 , piece_id=P645R4U4WSCWJR5XZSGFGVVF3QHAVSP6SN46CPQGBT6JWJB57ZIQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:21.818474194 (497h13m21.689557239s) [seg_pos={0 0}, upload=436400]: Segment reached optimal threshold of 13 pieces.
Dec 26 08:42:21.818682150 (497h13m21.689765195s) [noise=true , address=77.90.31.10:4517 , piece_id=RD3ARVUTAOQRS3YUQ7WL2LF47CXQFSVDDJSJUAEZAFVGUU4HJ7KQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF
Dec 26 08:42:21.818690926 (497h13m21.689773971s) [noise=true , address=77.90.31.10:4517 , piece_id=RD3ARVUTAOQRS3YUQ7WL2LF47CXQFSVDDJSJUAEZAFVGUU4HJ7KQ, seg_pos={0 0}, upload=436400]: Long tail cancellation due to optimal threshold
Dec 26 08:42:21.818911596 (497h13m21.689994631s) [noise=true , address=145.79.227.40:4051 , piece_id=NLPGNO63SJLYA65HYN2CJBJFS6OTLX7ISULHSM6Q54PWOOTVAVWQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF
Dec 26 08:42:21.818918088 (497h13m21.690001123s) [noise=true , address=145.79.227.40:4051 , piece_id=NLPGNO63SJLYA65HYN2CJBJFS6OTLX7ISULHSM6Q54PWOOTVAVWQ, seg_pos={0 0}, upload=436400]: Long tail cancellation due to optimal threshold
Dec 26 08:42:21.818941363 (497h13m21.690024418s) [noise=true , address=77.90.31.18:4808 , piece_id=INW543LSVQLUQQ4FZZXVTVPV2LSGPKMUVHCSKE3OQ2CQAIEQFD2Q, seg_pos={0 0}, upload=436400]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: expected piece hash; context canceled; EOF
Dec 26 08:42:21.818997390 (497h13m21.690080425s) [noise=true , address=77.90.31.18:4808 , piece_id=INW543LSVQLUQQ4FZZXVTVPV2LSGPKMUVHCSKE3OQ2CQAIEQFD2Q, seg_pos={0 0}, upload=436400]: Long tail cancellation due to optimal threshold
Dec 26 08:42:21.819032707 (497h13m21.690115742s) [seg_pos={0 0}, upload=436400]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 26 08:42:21.819224622 (497h13m21.690307657s) [upload=436400]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 08:42:21.819232357 (497h13m21.690315392s) [upload=436400]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 08:42:22.109308312 (497h13m21.980391357s) [upload=436400]: Done uploading segments...
Dec 26 08:42:22.109331417 (497h13m21.980414452s) [upload=436400]: Upload finished. err: <nil>
Duration: 5.923204416s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [........X...................................................] duration:
6.914716237s trace:
https://jaeger.dev.storj.io/trace/4b8b74dd4906ed4
FAIL (2025-12-26 08:47:52 UTC)
orbiter warning: context deadline exceeded
Duration: 6.914745013s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [..........................................X.................] duration:
28.789867207s trace:
https://jaeger.dev.storj.io/trace/6820217d73ecf2a
FAIL (2025-12-26 08:47:50 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
ss=45.91.116.13:4174 , piece_id=RLKVQ3QJM4STACAOUTWA7STEIG4PITJRIB4YOI3GNTYMPA76IFHQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.705106031 (497h3m36.576189076s) [noise=true , address=77.90.31.19:4870 , piece_id=ZHJNUA7P36RBWXQXMDLFRIHQPSZNOPGL67AQVDTDEPCRY2R4EMWQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.705493759 (497h3m36.576576804s) [noise=true , address=145.79.227.20:4443 , piece_id=HAFVT75GMRHFRR5E5VD4AOJFHGRUS56MQX3HNHI6FQYYMZT4J4FQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.705630279 (497h3m36.576713334s) [noise=true , address=77.90.31.15:4717 , piece_id=CVL5BLBIJ5DWSL7ZSEJ4I62RENOFPNWWNZCSE62CTMWGSS2XTE2Q, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.705923348 (497h3m36.577006393s) [noise=true , address=77.90.31.12:4607 , piece_id=TPFAMNN2BEYPSYRYOMDMSS77ZRB35HOJYWMJGTJEERRWGV5KQHHA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706055580 (497h3m36.577138615s) [noise=true , address=64.49.13.169:4705 , piece_id=NG53QDRTA5IAV5W64W6MSZVNCWDUDHBB3DP4BF6G7DTF5DAF7ECA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706362764 (497h3m36.577445819s) [noise=true , address=77.90.31.11:4581 , piece_id=NQOGQYJLH7W7CXFLBDW3XUN6Y6YVQEWAFBMLNKNHKTU5VCV4V3GQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706507651 (497h3m36.577590686s) [noise=true , address=145.79.227.44:4226 , piece_id=XFE2F2254TY5CMO7J5O6S3RDKHOHSLQCUGDNHB37VKOXPM63GJEA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706728992 (497h3m36.577812027s) [noise=true , address=45.91.116.4:4942 , piece_id=VDKHG53ATFYJDQYDBXKZ7SN2RNAAZBYGQVGAF34PCG4UYO7XMFGA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706869399 (497h3m36.577952444s) [noise=true , address=77.90.31.16:4750 , piece_id=HIZWD4OHAZ6UKMODPFHABQRQW7ESEZPVMSD45RI5HGXPLHDJLJHQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707077395 (497h3m36.57816044s) [noise=true , address=145.79.227.34:4909 , piece_id=TSBSOK42K3BBLLTUBV6UFWW33FK67FZHJZTKEQFAAQ7ITKY3RAEQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707229524 (497h3m36.578312559s) [noise=true , address=77.90.31.13:4655 , piece_id=B6QS3X24QQRPWA3O5ENXR32XJOJZBZZOVVDMSESJYKU3XBIPGDIQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707443251 (497h3m36.578526296s) [noise=true , address=145.79.227.5:4241 , piece_id=ANDNWFXV5YWOH55WDLOMJXSKZVN45XSIYT2I4QB52DNO2PT3HYKQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707633815 (497h3m36.57871685s) [noise=true , address=64.49.13.154:4930 , piece_id=IHQYIUPLXMAKL4S76Y6NAGDP2ZWFRQBOXXJJENZ6JRGBXWLKDCFQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707813797 (497h3m36.578896832s) [noise=true , address=145.79.227.13:4822 , piece_id=HSHVLD6IYQOG3KQYCZGENM4ZYH7XCKGRAZLEL6N2WELIBI2LNHSA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.708022374 (497h3m36.579105439s) [noise=true , address=145.79.227.30:4775 , piece_id=OG65SPMQI3XHNJHUHKSJ4NZKGAB764OYFE7D5CBJCP3YRAGYANQA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.708682712 (497h3m36.579765747s) [noise=true , address=77.90.31.17:4788 , piece_id=CIK5XZI3QMG7LFEOKBPZGNLDOORNP6IQV3QYSHLIY5XSF3LLXBBQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:38.013251430 (497h3m37.884334475s) [upload=436256]: Next returned nil segment
Dec 26 08:32:38.013277480 (497h3m37.884360525s) [upload=436256]: Waiting for error group managing segments...
Dec 26 08:32:38.345076384 (497h3m38.216159419s) [noise=true , address=64.49.13.154:4930 , piece_id=IHQYIUPLXMAKL4S76Y6NAGDP2ZWFRQBOXXJJENZ6JRGBXWLKDCFQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.593234781 (497h3m38.464317836s) [noise=true , address=145.79.227.20:4443 , piece_id=HAFVT75GMRHFRR5E5VD4AOJFHGRUS56MQX3HNHI6FQYYMZT4J4FQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.593901300 (497h3m38.464984335s) [noise=true , address=145.79.227.5:4241 , piece_id=ANDNWFXV5YWOH55WDLOMJXSKZVN45XSIYT2I4QB52DNO2PT3HYKQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.594508065 (497h3m38.4655911s) [noise=true , address=145.79.227.13:4822 , piece_id=HSHVLD6IYQOG3KQYCZGENM4ZYH7XCKGRAZLEL6N2WELIBI2LNHSA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.636070587 (497h3m38.507153653s) [noise=true , address=145.79.227.34:4909 , piece_id=TSBSOK42K3BBLLTUBV6UFWW33FK67FZHJZTKEQFAAQ7ITKY3RAEQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.750629848 (497h3m38.621712883s) [noise=true , address=77.90.31.11:4581 , piece_id=NQOGQYJLH7W7CXFLBDW3XUN6Y6YVQEWAFBMLNKNHKTU5VCV4V3GQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.814987737 (497h3m38.686070782s) [noise=true , address=45.91.116.4:4942 , piece_id=VDKHG53ATFYJDQYDBXKZ7SN2RNAAZBYGQVGAF34PCG4UYO7XMFGA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.847327759 (497h3m38.718410794s) [noise=true , address=77.90.31.15:4717 , piece_id=CVL5BLBIJ5DWSL7ZSEJ4I62RENOFPNWWNZCSE62CTMWGSS2XTE2Q, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.902089014 (497h3m38.773172049s) [noise=true , address=45.91.116.13:4174 , piece_id=RLKVQ3QJM4STACAOUTWA7STEIG4PITJRIB4YOI3GNTYMPA76IFHQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.912841608 (497h3m38.783924644s) [noise=true , address=77.90.31.12:4607 , piece_id=TPFAMNN2BEYPSYRYOMDMSS77ZRB35HOJYWMJGTJEERRWGV5KQHHA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.070851326 (497h3m38.941934371s) [noise=true , address=77.90.31.16:4750 , piece_id=HIZWD4OHAZ6UKMODPFHABQRQW7ESEZPVMSD45RI5HGXPLHDJLJHQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.469018898 (497h3m39.340101933s) [noise=true , address=145.79.227.30:4775 , piece_id=OG65SPMQI3XHNJHUHKSJ4NZKGAB764OYFE7D5CBJCP3YRAGYANQA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.477948602 (497h3m39.349031637s) [noise=true , address=145.79.227.44:4226 , piece_id=XFE2F2254TY5CMO7J5O6S3RDKHOHSLQCUGDNHB37VKOXPM63GJEA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.534878005 (497h3m39.40596105s) [noise=true , address=77.90.31.17:4788 , piece_id=CIK5XZI3QMG7LFEOKBPZGNLDOORNP6IQV3QYSHLIY5XSF3LLXBBQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.534902853 (497h3m39.405985898s) [seg_pos={0 0}, upload=436256]: Segment reached optimal threshold of 14 pieces.
Dec 26 08:32:39.535131698 (497h3m39.406214743s) [noise=true , address=77.90.31.19:4870 , piece_id=ZHJNUA7P36RBWXQXMDLFRIHQPSZNOPGL67AQVDTDEPCRY2R4EMWQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hNVXEVe8DW8F8meERGWkxAazEZKLKBeprNn6e5C94eEoAMBaR): protocol: expected piece hash; context canceled; EOF
Dec 26 08:32:39.535140075 (497h3m39.40622311s) [noise=true , address=77.90.31.19:4870 , piece_id=ZHJNUA7P36RBWXQXMDLFRIHQPSZNOPGL67AQVDTDEPCRY2R4EMWQ, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold
Dec 26 08:32:39.535367978 (497h3m39.406451013s) [noise=true , address=77.90.31.18:4838 , piece_id=V5LDUMURFU4H43E2A5MTLWBI6ZJHVPYMUILBDV5N7ERELPLTIBAA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1if7vf1cNTBwAhb6owi6tuGLYqSfa3feWhsvGSevwuwykhz2RY): protocol: expected piece hash; context canceled; EOF
Dec 26 08:32:39.535375542 (497h3m39.406458577s) [noise=true , address=77.90.31.18:4838 , piece_id=V5LDUMURFU4H43E2A5MTLWBI6ZJHVPYMUILBDV5N7ERELPLTIBAA, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold
Dec 26 08:32:39.535565575 (497h3m39.40664861s) [noise=true , address=77.90.31.13:4655 , piece_id=B6QS3X24QQRPWA3O5ENXR32XJOJZBZZOVVDMSESJYKU3XBIPGDIQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1K7fQM1NVA1L7ukmDtyWh2DYWypUoj3GbreETZpPL6gXfXmZDD): protocol: expected piece hash; context canceled; EOF
Dec 26 08:32:39.535606041 (497h3m39.406689077s) [noise=true , address=77.90.31.13:4655 , piece_id=B6QS3X24QQRPWA3O5ENXR32XJOJZBZZOVVDMSESJYKU3XBIPGDIQ, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold
Dec 26 08:32:39.535707996 (497h3m39.406791031s) [noise=true , address=64.49.13.169:4705 , piece_id=NG53QDRTA5IAV5W64W6MSZVNCWDUDHBB3DP4BF6G7DTF5DAF7ECA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: read tcp 188.166.225.170:35726->64.49.13.169:4705: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 26 08:32:39.535715059 (497h3m39.406798094s) [noise=true , address=64.49.13.169:4705 , piece_id=NG53QDRTA5IAV5W64W6MSZVNCWDUDHBB3DP4BF6G7DTF5DAF7ECA, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold
Dec 26 08:32:39.535740878 (497h3m39.406823923s) [seg_pos={0 0}, upload=436256]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 26 08:32:39.535861157 (497h3m39.406944202s) [upload=436256]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 08:32:39.535867620 (497h3m39.406950655s) [upload=436256]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 08:32:39.833497059 (497h3m39.704580164s) [upload=436256]: Done uploading segments...
Dec 26 08:32:39.833522858 (497h3m39.704605903s) [upload=436256]: Upload finished. err: <nil>
Duration: 28.789896602s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
21 out of 60 is failed (threshold=3) [....X..X.XX.X.X..X..X.XXX.....X......X.X...X.XX..X.XXX......] duration:
6.083568346s trace:
https://jaeger.dev.storj.io/trace/3312cc3668ef2c5f
FAIL (2025-12-26 08:47:39 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 26 08:42:18.506798665 (497h13m18.37788171s) [upload=436400]: Starting upload
Dec 26 08:42:18.506881102 (497h13m18.377964137s) [upload=436400]: Done starting upload
Dec 26 08:42:18.506963669 (497h13m18.378046714s) [upload=436400]: Uploading segments...
Dec 26 08:42:18.507170413 (497h13m18.378253478s) [upload=436400]: Got next segment. Inline: false
Dec 26 08:42:18.507247920 (497h13m18.378330965s) [upload=436400]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 26 08:42:18.507255454 (497h13m18.378338499s) [upload=436400]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 26 08:42:18.754283459 (497h13m18.625366504s) [seg_pos={0 0}, upload=436400]: Begin upload segment...
Dec 26 08:42:18.754356328 (497h13m18.625439373s) [seg_pos={0 0}, upload=436400]: Done begin upload segment.
Dec 26 08:42:18.754391745 (497h13m18.62547479s) [noise=true , address=145.79.227.40:4051 , piece_id=NLPGNO63SJLYA65HYN2CJBJFS6OTLX7ISULHSM6Q54PWOOTVAVWQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.754466177 (497h13m18.625549222s) [noise=true , address=77.90.31.17:4773 , piece_id=P645R4U4WSCWJR5XZSGFGVVF3QHAVSP6SN46CPQGBT6JWJB57ZIQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.755567554 (497h13m18.626650599s) [noise=true , address=77.90.31.18:4808 , piece_id=INW543LSVQLUQQ4FZZXVTVPV2LSGPKMUVHCSKE3OQ2CQAIEQFD2Q, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.755910858 (497h13m18.626993893s) [noise=true , address=145.79.227.17:4970 , piece_id=JXWYFTTKYNB6BSPBKWBK4YPF2PNAGWJKXHJAOUFIJVVIKIVNPUJA, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.756229915 (497h13m18.62731296s) [noise=true , address=145.79.227.46:4603 , piece_id=7L43455SH4JVSXBJYL3WVMRCWFQZ2CEI6DQA7MUXDYSKPPKESY7A, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.756613886 (497h13m18.627696922s) [noise=true , address=145.79.227.16:4912 , piece_id=TIPMXPQDX4HVH2NUQ3FUFT6J34SNM5DT2NHDXCZO76MMGPEEEHMQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.757020902 (497h13m18.628103937s) [noise=true , address=145.79.227.38:4977 , piece_id=MYMT3XBOXWU5DZOTPVHSVVXOKNHW7WWVCBECIO37VAYUTFK7XZSQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.757428849 (497h13m18.628511904s) [noise=true , address=77.90.31.11:4555 , piece_id=ZTSV63NMIHDFU464BAUD3MH3645MJOCGLTDQEKIRPD47EMLLO5EA, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.757902380 (497h13m18.628985425s) [noise=true , address=145.79.227.45:4229 , piece_id=GNXPJKQLV6ITICMBLE4PY2ODOKEJ6WWBUAPA74AO4IBVOVVEFSLA, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.757979276 (497h13m18.629062321s) [noise=true , address=77.90.31.21:4878 , piece_id=RNZ3EK2ZERZY6IQEBAZPHEUDBHNF7BARJF5FVZMLVVMLH5OK7WCQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.758365802 (497h13m18.629448837s) [noise=true , address=45.91.116.4:4937 , piece_id=7HT3WA63RZNVO3AX775OL37N4BJRXZCZ5P6A2A2Z6NPZX6RASSSQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.758773228 (497h13m18.629856263s) [noise=true , address=77.90.31.10:4517 , piece_id=RD3ARVUTAOQRS3YUQ7WL2LF47CXQFSVDDJSJUAEZAFVGUU4HJ7KQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.759342462 (497h13m18.630425507s) [noise=true , address=145.79.227.37:4567 , piece_id=J7Z3543ZQ2TIGFFHBO7V7GJKTGOY54Z6WAPYDI7ALYRYDS6NEK2Q, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.760026635 (497h13m18.63110968s) [noise=true , address=145.79.227.48:4301 , piece_id=IS2W2WAK7NOMKFEHY3QK7KS35XPQYC2723Q2ZJ27HTX4FPBQYHDQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.760798124 (497h13m18.631881169s) [noise=true , address=145.79.227.51:4411 , piece_id=7JDMY4F74SKL6NSFN2HO36EGUCM5MWVWR5HLXACHBLZXTDRRWI3Q, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.761677319 (497h13m18.632760364s) [noise=true , address=77.90.31.14:4664 , piece_id=JFEBDNYO2OQRCKCLBKWS43SZCZ7L2LSIF5FP2JS37FFARUEHCESA, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:20.466851608 (497h13m20.337934643s) [upload=436400]: Next returned nil segment
Dec 26 08:42:20.466865273 (497h13m20.337948318s) [upload=436400]: Waiting for error group managing segments...
Dec 26 08:42:20.750402218 (497h13m20.621485253s) [noise=true , address=145.79.227.37:4567 , piece_id=J7Z3543ZQ2TIGFFHBO7V7GJKTGOY54Z6WAPYDI7ALYRYDS6NEK2Q, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:20.763579740 (497h13m20.634662785s) [noise=true , address=145.79.227.48:4301 , piece_id=IS2W2WAK7NOMKFEHY3QK7KS35XPQYC2723Q2ZJ27HTX4FPBQYHDQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:20.767844120 (497h13m20.638927155s) [noise=true , address=145.79.227.38:4977 , piece_id=MYMT3XBOXWU5DZOTPVHSVVXOKNHW7WWVCBECIO37VAYUTFK7XZSQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:20.771941653 (497h13m20.643024698s) [noise=true , address=145.79.227.16:4912 , piece_id=TIPMXPQDX4HVH2NUQ3FUFT6J34SNM5DT2NHDXCZO76MMGPEEEHMQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:20.773887688 (497h13m20.644970723s) [noise=true , address=145.79.227.17:4970 , piece_id=JXWYFTTKYNB6BSPBKWBK4YPF2PNAGWJKXHJAOUFIJVVIKIVNPUJA, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:20.781414641 (497h13m20.652497676s) [noise=true , address=145.79.227.51:4411 , piece_id=7JDMY4F74SKL6NSFN2HO36EGUCM5MWVWR5HLXACHBLZXTDRRWI3Q, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:20.781591407 (497h13m20.652674452s) [noise=true , address=145.79.227.46:4603 , piece_id=7L43455SH4JVSXBJYL3WVMRCWFQZ2CEI6DQA7MUXDYSKPPKESY7A, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:20.787415879 (497h13m20.658498924s) [noise=true , address=145.79.227.45:4229 , piece_id=GNXPJKQLV6ITICMBLE4PY2ODOKEJ6WWBUAPA74AO4IBVOVVEFSLA, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:21.029225186 (497h13m20.900308221s) [noise=true , address=45.91.116.4:4937 , piece_id=7HT3WA63RZNVO3AX775OL37N4BJRXZCZ5P6A2A2Z6NPZX6RASSSQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:21.124342636 (497h13m20.995425671s) [noise=true , address=77.90.31.21:4878 , piece_id=RNZ3EK2ZERZY6IQEBAZPHEUDBHNF7BARJF5FVZMLVVMLH5OK7WCQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:21.148939092 (497h13m21.020022127s) [noise=true , address=77.90.31.14:4664 , piece_id=JFEBDNYO2OQRCKCLBKWS43SZCZ7L2LSIF5FP2JS37FFARUEHCESA, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:21.733228030 (497h13m21.604311075s) [noise=true , address=77.90.31.11:4555 , piece_id=ZTSV63NMIHDFU464BAUD3MH3645MJOCGLTDQEKIRPD47EMLLO5EA, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:21.818455067 (497h13m21.689538102s) [noise=true , address=77.90.31.17:4773 , piece_id=P645R4U4WSCWJR5XZSGFGVVF3QHAVSP6SN46CPQGBT6JWJB57ZIQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:21.818474194 (497h13m21.689557239s) [seg_pos={0 0}, upload=436400]: Segment reached optimal threshold of 13 pieces.
Dec 26 08:42:21.818682150 (497h13m21.689765195s) [noise=true , address=77.90.31.10:4517 , piece_id=RD3ARVUTAOQRS3YUQ7WL2LF47CXQFSVDDJSJUAEZAFVGUU4HJ7KQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF
Dec 26 08:42:21.818690926 (497h13m21.689773971s) [noise=true , address=77.90.31.10:4517 , piece_id=RD3ARVUTAOQRS3YUQ7WL2LF47CXQFSVDDJSJUAEZAFVGUU4HJ7KQ, seg_pos={0 0}, upload=436400]: Long tail cancellation due to optimal threshold
Dec 26 08:42:21.818911596 (497h13m21.689994631s) [noise=true , address=145.79.227.40:4051 , piece_id=NLPGNO63SJLYA65HYN2CJBJFS6OTLX7ISULHSM6Q54PWOOTVAVWQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF
Dec 26 08:42:21.818918088 (497h13m21.690001123s) [noise=true , address=145.79.227.40:4051 , piece_id=NLPGNO63SJLYA65HYN2CJBJFS6OTLX7ISULHSM6Q54PWOOTVAVWQ, seg_pos={0 0}, upload=436400]: Long tail cancellation due to optimal threshold
Dec 26 08:42:21.818941363 (497h13m21.690024418s) [noise=true , address=77.90.31.18:4808 , piece_id=INW543LSVQLUQQ4FZZXVTVPV2LSGPKMUVHCSKE3OQ2CQAIEQFD2Q, seg_pos={0 0}, upload=436400]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: expected piece hash; context canceled; EOF
Dec 26 08:42:21.818997390 (497h13m21.690080425s) [noise=true , address=77.90.31.18:4808 , piece_id=INW543LSVQLUQQ4FZZXVTVPV2LSGPKMUVHCSKE3OQ2CQAIEQFD2Q, seg_pos={0 0}, upload=436400]: Long tail cancellation due to optimal threshold
Dec 26 08:42:21.819032707 (497h13m21.690115742s) [seg_pos={0 0}, upload=436400]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 26 08:42:21.819224622 (497h13m21.690307657s) [upload=436400]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 08:42:21.819232357 (497h13m21.690315392s) [upload=436400]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 08:42:22.109308312 (497h13m21.980391357s) [upload=436400]: Done uploading segments...
Dec 26 08:42:22.109331417 (497h13m21.980414452s) [upload=436400]: Upload finished. err: <nil>
Duration: 6.083617299s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.........X..................................................] duration:
6.374066566s trace:
https://jaeger.dev.storj.io/trace/4b8b74dd4906ed4
FAIL (2025-12-26 08:47:27 UTC)
orbiter warning: context deadline exceeded
Duration: 6.374085432s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........................................X................] duration:
33.515368337s trace:
https://jaeger.dev.storj.io/trace/6820217d73ecf2a
FAIL (2025-12-26 08:47:14 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
ss=45.91.116.13:4174 , piece_id=RLKVQ3QJM4STACAOUTWA7STEIG4PITJRIB4YOI3GNTYMPA76IFHQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.705106031 (497h3m36.576189076s) [noise=true , address=77.90.31.19:4870 , piece_id=ZHJNUA7P36RBWXQXMDLFRIHQPSZNOPGL67AQVDTDEPCRY2R4EMWQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.705493759 (497h3m36.576576804s) [noise=true , address=145.79.227.20:4443 , piece_id=HAFVT75GMRHFRR5E5VD4AOJFHGRUS56MQX3HNHI6FQYYMZT4J4FQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.705630279 (497h3m36.576713334s) [noise=true , address=77.90.31.15:4717 , piece_id=CVL5BLBIJ5DWSL7ZSEJ4I62RENOFPNWWNZCSE62CTMWGSS2XTE2Q, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.705923348 (497h3m36.577006393s) [noise=true , address=77.90.31.12:4607 , piece_id=TPFAMNN2BEYPSYRYOMDMSS77ZRB35HOJYWMJGTJEERRWGV5KQHHA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706055580 (497h3m36.577138615s) [noise=true , address=64.49.13.169:4705 , piece_id=NG53QDRTA5IAV5W64W6MSZVNCWDUDHBB3DP4BF6G7DTF5DAF7ECA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706362764 (497h3m36.577445819s) [noise=true , address=77.90.31.11:4581 , piece_id=NQOGQYJLH7W7CXFLBDW3XUN6Y6YVQEWAFBMLNKNHKTU5VCV4V3GQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706507651 (497h3m36.577590686s) [noise=true , address=145.79.227.44:4226 , piece_id=XFE2F2254TY5CMO7J5O6S3RDKHOHSLQCUGDNHB37VKOXPM63GJEA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706728992 (497h3m36.577812027s) [noise=true , address=45.91.116.4:4942 , piece_id=VDKHG53ATFYJDQYDBXKZ7SN2RNAAZBYGQVGAF34PCG4UYO7XMFGA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706869399 (497h3m36.577952444s) [noise=true , address=77.90.31.16:4750 , piece_id=HIZWD4OHAZ6UKMODPFHABQRQW7ESEZPVMSD45RI5HGXPLHDJLJHQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707077395 (497h3m36.57816044s) [noise=true , address=145.79.227.34:4909 , piece_id=TSBSOK42K3BBLLTUBV6UFWW33FK67FZHJZTKEQFAAQ7ITKY3RAEQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707229524 (497h3m36.578312559s) [noise=true , address=77.90.31.13:4655 , piece_id=B6QS3X24QQRPWA3O5ENXR32XJOJZBZZOVVDMSESJYKU3XBIPGDIQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707443251 (497h3m36.578526296s) [noise=true , address=145.79.227.5:4241 , piece_id=ANDNWFXV5YWOH55WDLOMJXSKZVN45XSIYT2I4QB52DNO2PT3HYKQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707633815 (497h3m36.57871685s) [noise=true , address=64.49.13.154:4930 , piece_id=IHQYIUPLXMAKL4S76Y6NAGDP2ZWFRQBOXXJJENZ6JRGBXWLKDCFQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707813797 (497h3m36.578896832s) [noise=true , address=145.79.227.13:4822 , piece_id=HSHVLD6IYQOG3KQYCZGENM4ZYH7XCKGRAZLEL6N2WELIBI2LNHSA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.708022374 (497h3m36.579105439s) [noise=true , address=145.79.227.30:4775 , piece_id=OG65SPMQI3XHNJHUHKSJ4NZKGAB764OYFE7D5CBJCP3YRAGYANQA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.708682712 (497h3m36.579765747s) [noise=true , address=77.90.31.17:4788 , piece_id=CIK5XZI3QMG7LFEOKBPZGNLDOORNP6IQV3QYSHLIY5XSF3LLXBBQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:38.013251430 (497h3m37.884334475s) [upload=436256]: Next returned nil segment
Dec 26 08:32:38.013277480 (497h3m37.884360525s) [upload=436256]: Waiting for error group managing segments...
Dec 26 08:32:38.345076384 (497h3m38.216159419s) [noise=true , address=64.49.13.154:4930 , piece_id=IHQYIUPLXMAKL4S76Y6NAGDP2ZWFRQBOXXJJENZ6JRGBXWLKDCFQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.593234781 (497h3m38.464317836s) [noise=true , address=145.79.227.20:4443 , piece_id=HAFVT75GMRHFRR5E5VD4AOJFHGRUS56MQX3HNHI6FQYYMZT4J4FQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.593901300 (497h3m38.464984335s) [noise=true , address=145.79.227.5:4241 , piece_id=ANDNWFXV5YWOH55WDLOMJXSKZVN45XSIYT2I4QB52DNO2PT3HYKQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.594508065 (497h3m38.4655911s) [noise=true , address=145.79.227.13:4822 , piece_id=HSHVLD6IYQOG3KQYCZGENM4ZYH7XCKGRAZLEL6N2WELIBI2LNHSA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.636070587 (497h3m38.507153653s) [noise=true , address=145.79.227.34:4909 , piece_id=TSBSOK42K3BBLLTUBV6UFWW33FK67FZHJZTKEQFAAQ7ITKY3RAEQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.750629848 (497h3m38.621712883s) [noise=true , address=77.90.31.11:4581 , piece_id=NQOGQYJLH7W7CXFLBDW3XUN6Y6YVQEWAFBMLNKNHKTU5VCV4V3GQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.814987737 (497h3m38.686070782s) [noise=true , address=45.91.116.4:4942 , piece_id=VDKHG53ATFYJDQYDBXKZ7SN2RNAAZBYGQVGAF34PCG4UYO7XMFGA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.847327759 (497h3m38.718410794s) [noise=true , address=77.90.31.15:4717 , piece_id=CVL5BLBIJ5DWSL7ZSEJ4I62RENOFPNWWNZCSE62CTMWGSS2XTE2Q, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.902089014 (497h3m38.773172049s) [noise=true , address=45.91.116.13:4174 , piece_id=RLKVQ3QJM4STACAOUTWA7STEIG4PITJRIB4YOI3GNTYMPA76IFHQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.912841608 (497h3m38.783924644s) [noise=true , address=77.90.31.12:4607 , piece_id=TPFAMNN2BEYPSYRYOMDMSS77ZRB35HOJYWMJGTJEERRWGV5KQHHA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.070851326 (497h3m38.941934371s) [noise=true , address=77.90.31.16:4750 , piece_id=HIZWD4OHAZ6UKMODPFHABQRQW7ESEZPVMSD45RI5HGXPLHDJLJHQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.469018898 (497h3m39.340101933s) [noise=true , address=145.79.227.30:4775 , piece_id=OG65SPMQI3XHNJHUHKSJ4NZKGAB764OYFE7D5CBJCP3YRAGYANQA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.477948602 (497h3m39.349031637s) [noise=true , address=145.79.227.44:4226 , piece_id=XFE2F2254TY5CMO7J5O6S3RDKHOHSLQCUGDNHB37VKOXPM63GJEA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.534878005 (497h3m39.40596105s) [noise=true , address=77.90.31.17:4788 , piece_id=CIK5XZI3QMG7LFEOKBPZGNLDOORNP6IQV3QYSHLIY5XSF3LLXBBQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.534902853 (497h3m39.405985898s) [seg_pos={0 0}, upload=436256]: Segment reached optimal threshold of 14 pieces.
Dec 26 08:32:39.535131698 (497h3m39.406214743s) [noise=true , address=77.90.31.19:4870 , piece_id=ZHJNUA7P36RBWXQXMDLFRIHQPSZNOPGL67AQVDTDEPCRY2R4EMWQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hNVXEVe8DW8F8meERGWkxAazEZKLKBeprNn6e5C94eEoAMBaR): protocol: expected piece hash; context canceled; EOF
Dec 26 08:32:39.535140075 (497h3m39.40622311s) [noise=true , address=77.90.31.19:4870 , piece_id=ZHJNUA7P36RBWXQXMDLFRIHQPSZNOPGL67AQVDTDEPCRY2R4EMWQ, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold
Dec 26 08:32:39.535367978 (497h3m39.406451013s) [noise=true , address=77.90.31.18:4838 , piece_id=V5LDUMURFU4H43E2A5MTLWBI6ZJHVPYMUILBDV5N7ERELPLTIBAA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1if7vf1cNTBwAhb6owi6tuGLYqSfa3feWhsvGSevwuwykhz2RY): protocol: expected piece hash; context canceled; EOF
Dec 26 08:32:39.535375542 (497h3m39.406458577s) [noise=true , address=77.90.31.18:4838 , piece_id=V5LDUMURFU4H43E2A5MTLWBI6ZJHVPYMUILBDV5N7ERELPLTIBAA, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold
Dec 26 08:32:39.535565575 (497h3m39.40664861s) [noise=true , address=77.90.31.13:4655 , piece_id=B6QS3X24QQRPWA3O5ENXR32XJOJZBZZOVVDMSESJYKU3XBIPGDIQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1K7fQM1NVA1L7ukmDtyWh2DYWypUoj3GbreETZpPL6gXfXmZDD): protocol: expected piece hash; context canceled; EOF
Dec 26 08:32:39.535606041 (497h3m39.406689077s) [noise=true , address=77.90.31.13:4655 , piece_id=B6QS3X24QQRPWA3O5ENXR32XJOJZBZZOVVDMSESJYKU3XBIPGDIQ, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold
Dec 26 08:32:39.535707996 (497h3m39.406791031s) [noise=true , address=64.49.13.169:4705 , piece_id=NG53QDRTA5IAV5W64W6MSZVNCWDUDHBB3DP4BF6G7DTF5DAF7ECA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: read tcp 188.166.225.170:35726->64.49.13.169:4705: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 26 08:32:39.535715059 (497h3m39.406798094s) [noise=true , address=64.49.13.169:4705 , piece_id=NG53QDRTA5IAV5W64W6MSZVNCWDUDHBB3DP4BF6G7DTF5DAF7ECA, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold
Dec 26 08:32:39.535740878 (497h3m39.406823923s) [seg_pos={0 0}, upload=436256]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 26 08:32:39.535861157 (497h3m39.406944202s) [upload=436256]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 08:32:39.535867620 (497h3m39.406950655s) [upload=436256]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 08:32:39.833497059 (497h3m39.704580164s) [upload=436256]: Done uploading segments...
Dec 26 08:32:39.833522858 (497h3m39.704605903s) [upload=436256]: Upload finished. err: <nil>
Duration: 33.515404345s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [.......X....................................................] duration:
3.485055565s trace:
https://jaeger.dev.storj.io/trace/185fddc4dc40eba9
FAIL (2025-12-26 08:47:11 UTC)
orbiter warning: context deadline exceeded
Duration: 3.485112874s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..........X.................................................] duration:
24.551190216s trace:
https://jaeger.dev.storj.io/trace/4b8b74dd4906ed4
FAIL (2025-12-26 08:47:03 UTC)
orbiter warning: context deadline exceeded
Duration: 24.551225031s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
21 out of 60 is failed (threshold=3) [.....X..X.XX.X.X..X..X.XXX.....X......X.X...X.XX..X.XXX.....] duration:
6.00516272s trace:
https://jaeger.dev.storj.io/trace/3312cc3668ef2c5f
FAIL (2025-12-26 08:46:46 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 26 08:42:18.506798665 (497h13m18.37788171s) [upload=436400]: Starting upload
Dec 26 08:42:18.506881102 (497h13m18.377964137s) [upload=436400]: Done starting upload
Dec 26 08:42:18.506963669 (497h13m18.378046714s) [upload=436400]: Uploading segments...
Dec 26 08:42:18.507170413 (497h13m18.378253478s) [upload=436400]: Got next segment. Inline: false
Dec 26 08:42:18.507247920 (497h13m18.378330965s) [upload=436400]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 26 08:42:18.507255454 (497h13m18.378338499s) [upload=436400]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 26 08:42:18.754283459 (497h13m18.625366504s) [seg_pos={0 0}, upload=436400]: Begin upload segment...
Dec 26 08:42:18.754356328 (497h13m18.625439373s) [seg_pos={0 0}, upload=436400]: Done begin upload segment.
Dec 26 08:42:18.754391745 (497h13m18.62547479s) [noise=true , address=145.79.227.40:4051 , piece_id=NLPGNO63SJLYA65HYN2CJBJFS6OTLX7ISULHSM6Q54PWOOTVAVWQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.754466177 (497h13m18.625549222s) [noise=true , address=77.90.31.17:4773 , piece_id=P645R4U4WSCWJR5XZSGFGVVF3QHAVSP6SN46CPQGBT6JWJB57ZIQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.755567554 (497h13m18.626650599s) [noise=true , address=77.90.31.18:4808 , piece_id=INW543LSVQLUQQ4FZZXVTVPV2LSGPKMUVHCSKE3OQ2CQAIEQFD2Q, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.755910858 (497h13m18.626993893s) [noise=true , address=145.79.227.17:4970 , piece_id=JXWYFTTKYNB6BSPBKWBK4YPF2PNAGWJKXHJAOUFIJVVIKIVNPUJA, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.756229915 (497h13m18.62731296s) [noise=true , address=145.79.227.46:4603 , piece_id=7L43455SH4JVSXBJYL3WVMRCWFQZ2CEI6DQA7MUXDYSKPPKESY7A, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.756613886 (497h13m18.627696922s) [noise=true , address=145.79.227.16:4912 , piece_id=TIPMXPQDX4HVH2NUQ3FUFT6J34SNM5DT2NHDXCZO76MMGPEEEHMQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.757020902 (497h13m18.628103937s) [noise=true , address=145.79.227.38:4977 , piece_id=MYMT3XBOXWU5DZOTPVHSVVXOKNHW7WWVCBECIO37VAYUTFK7XZSQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.757428849 (497h13m18.628511904s) [noise=true , address=77.90.31.11:4555 , piece_id=ZTSV63NMIHDFU464BAUD3MH3645MJOCGLTDQEKIRPD47EMLLO5EA, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.757902380 (497h13m18.628985425s) [noise=true , address=145.79.227.45:4229 , piece_id=GNXPJKQLV6ITICMBLE4PY2ODOKEJ6WWBUAPA74AO4IBVOVVEFSLA, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.757979276 (497h13m18.629062321s) [noise=true , address=77.90.31.21:4878 , piece_id=RNZ3EK2ZERZY6IQEBAZPHEUDBHNF7BARJF5FVZMLVVMLH5OK7WCQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.758365802 (497h13m18.629448837s) [noise=true , address=45.91.116.4:4937 , piece_id=7HT3WA63RZNVO3AX775OL37N4BJRXZCZ5P6A2A2Z6NPZX6RASSSQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.758773228 (497h13m18.629856263s) [noise=true , address=77.90.31.10:4517 , piece_id=RD3ARVUTAOQRS3YUQ7WL2LF47CXQFSVDDJSJUAEZAFVGUU4HJ7KQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.759342462 (497h13m18.630425507s) [noise=true , address=145.79.227.37:4567 , piece_id=J7Z3543ZQ2TIGFFHBO7V7GJKTGOY54Z6WAPYDI7ALYRYDS6NEK2Q, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.760026635 (497h13m18.63110968s) [noise=true , address=145.79.227.48:4301 , piece_id=IS2W2WAK7NOMKFEHY3QK7KS35XPQYC2723Q2ZJ27HTX4FPBQYHDQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.760798124 (497h13m18.631881169s) [noise=true , address=145.79.227.51:4411 , piece_id=7JDMY4F74SKL6NSFN2HO36EGUCM5MWVWR5HLXACHBLZXTDRRWI3Q, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.761677319 (497h13m18.632760364s) [noise=true , address=77.90.31.14:4664 , piece_id=JFEBDNYO2OQRCKCLBKWS43SZCZ7L2LSIF5FP2JS37FFARUEHCESA, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:20.466851608 (497h13m20.337934643s) [upload=436400]: Next returned nil segment
Dec 26 08:42:20.466865273 (497h13m20.337948318s) [upload=436400]: Waiting for error group managing segments...
Dec 26 08:42:20.750402218 (497h13m20.621485253s) [noise=true , address=145.79.227.37:4567 , piece_id=J7Z3543ZQ2TIGFFHBO7V7GJKTGOY54Z6WAPYDI7ALYRYDS6NEK2Q, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:20.763579740 (497h13m20.634662785s) [noise=true , address=145.79.227.48:4301 , piece_id=IS2W2WAK7NOMKFEHY3QK7KS35XPQYC2723Q2ZJ27HTX4FPBQYHDQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:20.767844120 (497h13m20.638927155s) [noise=true , address=145.79.227.38:4977 , piece_id=MYMT3XBOXWU5DZOTPVHSVVXOKNHW7WWVCBECIO37VAYUTFK7XZSQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:20.771941653 (497h13m20.643024698s) [noise=true , address=145.79.227.16:4912 , piece_id=TIPMXPQDX4HVH2NUQ3FUFT6J34SNM5DT2NHDXCZO76MMGPEEEHMQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:20.773887688 (497h13m20.644970723s) [noise=true , address=145.79.227.17:4970 , piece_id=JXWYFTTKYNB6BSPBKWBK4YPF2PNAGWJKXHJAOUFIJVVIKIVNPUJA, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:20.781414641 (497h13m20.652497676s) [noise=true , address=145.79.227.51:4411 , piece_id=7JDMY4F74SKL6NSFN2HO36EGUCM5MWVWR5HLXACHBLZXTDRRWI3Q, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:20.781591407 (497h13m20.652674452s) [noise=true , address=145.79.227.46:4603 , piece_id=7L43455SH4JVSXBJYL3WVMRCWFQZ2CEI6DQA7MUXDYSKPPKESY7A, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:20.787415879 (497h13m20.658498924s) [noise=true , address=145.79.227.45:4229 , piece_id=GNXPJKQLV6ITICMBLE4PY2ODOKEJ6WWBUAPA74AO4IBVOVVEFSLA, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:21.029225186 (497h13m20.900308221s) [noise=true , address=45.91.116.4:4937 , piece_id=7HT3WA63RZNVO3AX775OL37N4BJRXZCZ5P6A2A2Z6NPZX6RASSSQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:21.124342636 (497h13m20.995425671s) [noise=true , address=77.90.31.21:4878 , piece_id=RNZ3EK2ZERZY6IQEBAZPHEUDBHNF7BARJF5FVZMLVVMLH5OK7WCQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:21.148939092 (497h13m21.020022127s) [noise=true , address=77.90.31.14:4664 , piece_id=JFEBDNYO2OQRCKCLBKWS43SZCZ7L2LSIF5FP2JS37FFARUEHCESA, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:21.733228030 (497h13m21.604311075s) [noise=true , address=77.90.31.11:4555 , piece_id=ZTSV63NMIHDFU464BAUD3MH3645MJOCGLTDQEKIRPD47EMLLO5EA, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:21.818455067 (497h13m21.689538102s) [noise=true , address=77.90.31.17:4773 , piece_id=P645R4U4WSCWJR5XZSGFGVVF3QHAVSP6SN46CPQGBT6JWJB57ZIQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:21.818474194 (497h13m21.689557239s) [seg_pos={0 0}, upload=436400]: Segment reached optimal threshold of 13 pieces.
Dec 26 08:42:21.818682150 (497h13m21.689765195s) [noise=true , address=77.90.31.10:4517 , piece_id=RD3ARVUTAOQRS3YUQ7WL2LF47CXQFSVDDJSJUAEZAFVGUU4HJ7KQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF
Dec 26 08:42:21.818690926 (497h13m21.689773971s) [noise=true , address=77.90.31.10:4517 , piece_id=RD3ARVUTAOQRS3YUQ7WL2LF47CXQFSVDDJSJUAEZAFVGUU4HJ7KQ, seg_pos={0 0}, upload=436400]: Long tail cancellation due to optimal threshold
Dec 26 08:42:21.818911596 (497h13m21.689994631s) [noise=true , address=145.79.227.40:4051 , piece_id=NLPGNO63SJLYA65HYN2CJBJFS6OTLX7ISULHSM6Q54PWOOTVAVWQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF
Dec 26 08:42:21.818918088 (497h13m21.690001123s) [noise=true , address=145.79.227.40:4051 , piece_id=NLPGNO63SJLYA65HYN2CJBJFS6OTLX7ISULHSM6Q54PWOOTVAVWQ, seg_pos={0 0}, upload=436400]: Long tail cancellation due to optimal threshold
Dec 26 08:42:21.818941363 (497h13m21.690024418s) [noise=true , address=77.90.31.18:4808 , piece_id=INW543LSVQLUQQ4FZZXVTVPV2LSGPKMUVHCSKE3OQ2CQAIEQFD2Q, seg_pos={0 0}, upload=436400]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: expected piece hash; context canceled; EOF
Dec 26 08:42:21.818997390 (497h13m21.690080425s) [noise=true , address=77.90.31.18:4808 , piece_id=INW543LSVQLUQQ4FZZXVTVPV2LSGPKMUVHCSKE3OQ2CQAIEQFD2Q, seg_pos={0 0}, upload=436400]: Long tail cancellation due to optimal threshold
Dec 26 08:42:21.819032707 (497h13m21.690115742s) [seg_pos={0 0}, upload=436400]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 26 08:42:21.819224622 (497h13m21.690307657s) [upload=436400]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 08:42:21.819232357 (497h13m21.690315392s) [upload=436400]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 08:42:22.109308312 (497h13m21.980391357s) [upload=436400]: Done uploading segments...
Dec 26 08:42:22.109331417 (497h13m21.980414452s) [upload=436400]: Upload finished. err: <nil>
Duration: 6.005189811s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........X................................................] duration:
5.913691301s trace:
https://jaeger.dev.storj.io/trace/4b8b74dd4906ed4
FAIL (2025-12-26 08:46:13 UTC)
orbiter warning: context deadline exceeded
Duration: 5.913745214s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [........X...................................................] duration:
3.436996221s trace:
https://jaeger.dev.storj.io/trace/185fddc4dc40eba9
FAIL (2025-12-26 08:46:11 UTC)
orbiter warning: context deadline exceeded
Duration: 3.437019425s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [.........X..................................................] duration:
3.472986594s trace:
https://jaeger.dev.storj.io/trace/185fddc4dc40eba9
FAIL (2025-12-26 08:45:56 UTC)
orbiter warning: context deadline exceeded
Duration: 3.473013094s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [............................................X...............] duration:
23.484945709s trace:
https://jaeger.dev.storj.io/trace/6820217d73ecf2a
FAIL (2025-12-26 08:45:55 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
ss=45.91.116.13:4174 , piece_id=RLKVQ3QJM4STACAOUTWA7STEIG4PITJRIB4YOI3GNTYMPA76IFHQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.705106031 (497h3m36.576189076s) [noise=true , address=77.90.31.19:4870 , piece_id=ZHJNUA7P36RBWXQXMDLFRIHQPSZNOPGL67AQVDTDEPCRY2R4EMWQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.705493759 (497h3m36.576576804s) [noise=true , address=145.79.227.20:4443 , piece_id=HAFVT75GMRHFRR5E5VD4AOJFHGRUS56MQX3HNHI6FQYYMZT4J4FQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.705630279 (497h3m36.576713334s) [noise=true , address=77.90.31.15:4717 , piece_id=CVL5BLBIJ5DWSL7ZSEJ4I62RENOFPNWWNZCSE62CTMWGSS2XTE2Q, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.705923348 (497h3m36.577006393s) [noise=true , address=77.90.31.12:4607 , piece_id=TPFAMNN2BEYPSYRYOMDMSS77ZRB35HOJYWMJGTJEERRWGV5KQHHA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706055580 (497h3m36.577138615s) [noise=true , address=64.49.13.169:4705 , piece_id=NG53QDRTA5IAV5W64W6MSZVNCWDUDHBB3DP4BF6G7DTF5DAF7ECA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706362764 (497h3m36.577445819s) [noise=true , address=77.90.31.11:4581 , piece_id=NQOGQYJLH7W7CXFLBDW3XUN6Y6YVQEWAFBMLNKNHKTU5VCV4V3GQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706507651 (497h3m36.577590686s) [noise=true , address=145.79.227.44:4226 , piece_id=XFE2F2254TY5CMO7J5O6S3RDKHOHSLQCUGDNHB37VKOXPM63GJEA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706728992 (497h3m36.577812027s) [noise=true , address=45.91.116.4:4942 , piece_id=VDKHG53ATFYJDQYDBXKZ7SN2RNAAZBYGQVGAF34PCG4UYO7XMFGA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706869399 (497h3m36.577952444s) [noise=true , address=77.90.31.16:4750 , piece_id=HIZWD4OHAZ6UKMODPFHABQRQW7ESEZPVMSD45RI5HGXPLHDJLJHQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707077395 (497h3m36.57816044s) [noise=true , address=145.79.227.34:4909 , piece_id=TSBSOK42K3BBLLTUBV6UFWW33FK67FZHJZTKEQFAAQ7ITKY3RAEQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707229524 (497h3m36.578312559s) [noise=true , address=77.90.31.13:4655 , piece_id=B6QS3X24QQRPWA3O5ENXR32XJOJZBZZOVVDMSESJYKU3XBIPGDIQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707443251 (497h3m36.578526296s) [noise=true , address=145.79.227.5:4241 , piece_id=ANDNWFXV5YWOH55WDLOMJXSKZVN45XSIYT2I4QB52DNO2PT3HYKQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707633815 (497h3m36.57871685s) [noise=true , address=64.49.13.154:4930 , piece_id=IHQYIUPLXMAKL4S76Y6NAGDP2ZWFRQBOXXJJENZ6JRGBXWLKDCFQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707813797 (497h3m36.578896832s) [noise=true , address=145.79.227.13:4822 , piece_id=HSHVLD6IYQOG3KQYCZGENM4ZYH7XCKGRAZLEL6N2WELIBI2LNHSA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.708022374 (497h3m36.579105439s) [noise=true , address=145.79.227.30:4775 , piece_id=OG65SPMQI3XHNJHUHKSJ4NZKGAB764OYFE7D5CBJCP3YRAGYANQA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.708682712 (497h3m36.579765747s) [noise=true , address=77.90.31.17:4788 , piece_id=CIK5XZI3QMG7LFEOKBPZGNLDOORNP6IQV3QYSHLIY5XSF3LLXBBQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:38.013251430 (497h3m37.884334475s) [upload=436256]: Next returned nil segment
Dec 26 08:32:38.013277480 (497h3m37.884360525s) [upload=436256]: Waiting for error group managing segments...
Dec 26 08:32:38.345076384 (497h3m38.216159419s) [noise=true , address=64.49.13.154:4930 , piece_id=IHQYIUPLXMAKL4S76Y6NAGDP2ZWFRQBOXXJJENZ6JRGBXWLKDCFQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.593234781 (497h3m38.464317836s) [noise=true , address=145.79.227.20:4443 , piece_id=HAFVT75GMRHFRR5E5VD4AOJFHGRUS56MQX3HNHI6FQYYMZT4J4FQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.593901300 (497h3m38.464984335s) [noise=true , address=145.79.227.5:4241 , piece_id=ANDNWFXV5YWOH55WDLOMJXSKZVN45XSIYT2I4QB52DNO2PT3HYKQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.594508065 (497h3m38.4655911s) [noise=true , address=145.79.227.13:4822 , piece_id=HSHVLD6IYQOG3KQYCZGENM4ZYH7XCKGRAZLEL6N2WELIBI2LNHSA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.636070587 (497h3m38.507153653s) [noise=true , address=145.79.227.34:4909 , piece_id=TSBSOK42K3BBLLTUBV6UFWW33FK67FZHJZTKEQFAAQ7ITKY3RAEQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.750629848 (497h3m38.621712883s) [noise=true , address=77.90.31.11:4581 , piece_id=NQOGQYJLH7W7CXFLBDW3XUN6Y6YVQEWAFBMLNKNHKTU5VCV4V3GQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.814987737 (497h3m38.686070782s) [noise=true , address=45.91.116.4:4942 , piece_id=VDKHG53ATFYJDQYDBXKZ7SN2RNAAZBYGQVGAF34PCG4UYO7XMFGA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.847327759 (497h3m38.718410794s) [noise=true , address=77.90.31.15:4717 , piece_id=CVL5BLBIJ5DWSL7ZSEJ4I62RENOFPNWWNZCSE62CTMWGSS2XTE2Q, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.902089014 (497h3m38.773172049s) [noise=true , address=45.91.116.13:4174 , piece_id=RLKVQ3QJM4STACAOUTWA7STEIG4PITJRIB4YOI3GNTYMPA76IFHQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.912841608 (497h3m38.783924644s) [noise=true , address=77.90.31.12:4607 , piece_id=TPFAMNN2BEYPSYRYOMDMSS77ZRB35HOJYWMJGTJEERRWGV5KQHHA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.070851326 (497h3m38.941934371s) [noise=true , address=77.90.31.16:4750 , piece_id=HIZWD4OHAZ6UKMODPFHABQRQW7ESEZPVMSD45RI5HGXPLHDJLJHQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.469018898 (497h3m39.340101933s) [noise=true , address=145.79.227.30:4775 , piece_id=OG65SPMQI3XHNJHUHKSJ4NZKGAB764OYFE7D5CBJCP3YRAGYANQA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.477948602 (497h3m39.349031637s) [noise=true , address=145.79.227.44:4226 , piece_id=XFE2F2254TY5CMO7J5O6S3RDKHOHSLQCUGDNHB37VKOXPM63GJEA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.534878005 (497h3m39.40596105s) [noise=true , address=77.90.31.17:4788 , piece_id=CIK5XZI3QMG7LFEOKBPZGNLDOORNP6IQV3QYSHLIY5XSF3LLXBBQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.534902853 (497h3m39.405985898s) [seg_pos={0 0}, upload=436256]: Segment reached optimal threshold of 14 pieces.
Dec 26 08:32:39.535131698 (497h3m39.406214743s) [noise=true , address=77.90.31.19:4870 , piece_id=ZHJNUA7P36RBWXQXMDLFRIHQPSZNOPGL67AQVDTDEPCRY2R4EMWQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hNVXEVe8DW8F8meERGWkxAazEZKLKBeprNn6e5C94eEoAMBaR): protocol: expected piece hash; context canceled; EOF
Dec 26 08:32:39.535140075 (497h3m39.40622311s) [noise=true , address=77.90.31.19:4870 , piece_id=ZHJNUA7P36RBWXQXMDLFRIHQPSZNOPGL67AQVDTDEPCRY2R4EMWQ, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold
Dec 26 08:32:39.535367978 (497h3m39.406451013s) [noise=true , address=77.90.31.18:4838 , piece_id=V5LDUMURFU4H43E2A5MTLWBI6ZJHVPYMUILBDV5N7ERELPLTIBAA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1if7vf1cNTBwAhb6owi6tuGLYqSfa3feWhsvGSevwuwykhz2RY): protocol: expected piece hash; context canceled; EOF
Dec 26 08:32:39.535375542 (497h3m39.406458577s) [noise=true , address=77.90.31.18:4838 , piece_id=V5LDUMURFU4H43E2A5MTLWBI6ZJHVPYMUILBDV5N7ERELPLTIBAA, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold
Dec 26 08:32:39.535565575 (497h3m39.40664861s) [noise=true , address=77.90.31.13:4655 , piece_id=B6QS3X24QQRPWA3O5ENXR32XJOJZBZZOVVDMSESJYKU3XBIPGDIQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1K7fQM1NVA1L7ukmDtyWh2DYWypUoj3GbreETZpPL6gXfXmZDD): protocol: expected piece hash; context canceled; EOF
Dec 26 08:32:39.535606041 (497h3m39.406689077s) [noise=true , address=77.90.31.13:4655 , piece_id=B6QS3X24QQRPWA3O5ENXR32XJOJZBZZOVVDMSESJYKU3XBIPGDIQ, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold
Dec 26 08:32:39.535707996 (497h3m39.406791031s) [noise=true , address=64.49.13.169:4705 , piece_id=NG53QDRTA5IAV5W64W6MSZVNCWDUDHBB3DP4BF6G7DTF5DAF7ECA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: read tcp 188.166.225.170:35726->64.49.13.169:4705: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 26 08:32:39.535715059 (497h3m39.406798094s) [noise=true , address=64.49.13.169:4705 , piece_id=NG53QDRTA5IAV5W64W6MSZVNCWDUDHBB3DP4BF6G7DTF5DAF7ECA, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold
Dec 26 08:32:39.535740878 (497h3m39.406823923s) [seg_pos={0 0}, upload=436256]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 26 08:32:39.535861157 (497h3m39.406944202s) [upload=436256]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 08:32:39.535867620 (497h3m39.406950655s) [upload=436256]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 08:32:39.833497059 (497h3m39.704580164s) [upload=436256]: Done uploading segments...
Dec 26 08:32:39.833522858 (497h3m39.704605903s) [upload=436256]: Upload finished. err: <nil>
Duration: 23.484993821s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
21 out of 60 is failed (threshold=3) [......X..X.XX.X.X..X..X.XXX.....X......X.X...X.XX..X.XXX....] duration:
5.739274301s trace:
https://jaeger.dev.storj.io/trace/3312cc3668ef2c5f
FAIL (2025-12-26 08:45:43 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 26 08:42:18.506798665 (497h13m18.37788171s) [upload=436400]: Starting upload
Dec 26 08:42:18.506881102 (497h13m18.377964137s) [upload=436400]: Done starting upload
Dec 26 08:42:18.506963669 (497h13m18.378046714s) [upload=436400]: Uploading segments...
Dec 26 08:42:18.507170413 (497h13m18.378253478s) [upload=436400]: Got next segment. Inline: false
Dec 26 08:42:18.507247920 (497h13m18.378330965s) [upload=436400]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 26 08:42:18.507255454 (497h13m18.378338499s) [upload=436400]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 26 08:42:18.754283459 (497h13m18.625366504s) [seg_pos={0 0}, upload=436400]: Begin upload segment...
Dec 26 08:42:18.754356328 (497h13m18.625439373s) [seg_pos={0 0}, upload=436400]: Done begin upload segment.
Dec 26 08:42:18.754391745 (497h13m18.62547479s) [noise=true , address=145.79.227.40:4051 , piece_id=NLPGNO63SJLYA65HYN2CJBJFS6OTLX7ISULHSM6Q54PWOOTVAVWQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.754466177 (497h13m18.625549222s) [noise=true , address=77.90.31.17:4773 , piece_id=P645R4U4WSCWJR5XZSGFGVVF3QHAVSP6SN46CPQGBT6JWJB57ZIQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.755567554 (497h13m18.626650599s) [noise=true , address=77.90.31.18:4808 , piece_id=INW543LSVQLUQQ4FZZXVTVPV2LSGPKMUVHCSKE3OQ2CQAIEQFD2Q, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.755910858 (497h13m18.626993893s) [noise=true , address=145.79.227.17:4970 , piece_id=JXWYFTTKYNB6BSPBKWBK4YPF2PNAGWJKXHJAOUFIJVVIKIVNPUJA, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.756229915 (497h13m18.62731296s) [noise=true , address=145.79.227.46:4603 , piece_id=7L43455SH4JVSXBJYL3WVMRCWFQZ2CEI6DQA7MUXDYSKPPKESY7A, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.756613886 (497h13m18.627696922s) [noise=true , address=145.79.227.16:4912 , piece_id=TIPMXPQDX4HVH2NUQ3FUFT6J34SNM5DT2NHDXCZO76MMGPEEEHMQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.757020902 (497h13m18.628103937s) [noise=true , address=145.79.227.38:4977 , piece_id=MYMT3XBOXWU5DZOTPVHSVVXOKNHW7WWVCBECIO37VAYUTFK7XZSQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.757428849 (497h13m18.628511904s) [noise=true , address=77.90.31.11:4555 , piece_id=ZTSV63NMIHDFU464BAUD3MH3645MJOCGLTDQEKIRPD47EMLLO5EA, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.757902380 (497h13m18.628985425s) [noise=true , address=145.79.227.45:4229 , piece_id=GNXPJKQLV6ITICMBLE4PY2ODOKEJ6WWBUAPA74AO4IBVOVVEFSLA, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.757979276 (497h13m18.629062321s) [noise=true , address=77.90.31.21:4878 , piece_id=RNZ3EK2ZERZY6IQEBAZPHEUDBHNF7BARJF5FVZMLVVMLH5OK7WCQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.758365802 (497h13m18.629448837s) [noise=true , address=45.91.116.4:4937 , piece_id=7HT3WA63RZNVO3AX775OL37N4BJRXZCZ5P6A2A2Z6NPZX6RASSSQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.758773228 (497h13m18.629856263s) [noise=true , address=77.90.31.10:4517 , piece_id=RD3ARVUTAOQRS3YUQ7WL2LF47CXQFSVDDJSJUAEZAFVGUU4HJ7KQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.759342462 (497h13m18.630425507s) [noise=true , address=145.79.227.37:4567 , piece_id=J7Z3543ZQ2TIGFFHBO7V7GJKTGOY54Z6WAPYDI7ALYRYDS6NEK2Q, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.760026635 (497h13m18.63110968s) [noise=true , address=145.79.227.48:4301 , piece_id=IS2W2WAK7NOMKFEHY3QK7KS35XPQYC2723Q2ZJ27HTX4FPBQYHDQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.760798124 (497h13m18.631881169s) [noise=true , address=145.79.227.51:4411 , piece_id=7JDMY4F74SKL6NSFN2HO36EGUCM5MWVWR5HLXACHBLZXTDRRWI3Q, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.761677319 (497h13m18.632760364s) [noise=true , address=77.90.31.14:4664 , piece_id=JFEBDNYO2OQRCKCLBKWS43SZCZ7L2LSIF5FP2JS37FFARUEHCESA, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:20.466851608 (497h13m20.337934643s) [upload=436400]: Next returned nil segment
Dec 26 08:42:20.466865273 (497h13m20.337948318s) [upload=436400]: Waiting for error group managing segments...
Dec 26 08:42:20.750402218 (497h13m20.621485253s) [noise=true , address=145.79.227.37:4567 , piece_id=J7Z3543ZQ2TIGFFHBO7V7GJKTGOY54Z6WAPYDI7ALYRYDS6NEK2Q, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:20.763579740 (497h13m20.634662785s) [noise=true , address=145.79.227.48:4301 , piece_id=IS2W2WAK7NOMKFEHY3QK7KS35XPQYC2723Q2ZJ27HTX4FPBQYHDQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:20.767844120 (497h13m20.638927155s) [noise=true , address=145.79.227.38:4977 , piece_id=MYMT3XBOXWU5DZOTPVHSVVXOKNHW7WWVCBECIO37VAYUTFK7XZSQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:20.771941653 (497h13m20.643024698s) [noise=true , address=145.79.227.16:4912 , piece_id=TIPMXPQDX4HVH2NUQ3FUFT6J34SNM5DT2NHDXCZO76MMGPEEEHMQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:20.773887688 (497h13m20.644970723s) [noise=true , address=145.79.227.17:4970 , piece_id=JXWYFTTKYNB6BSPBKWBK4YPF2PNAGWJKXHJAOUFIJVVIKIVNPUJA, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:20.781414641 (497h13m20.652497676s) [noise=true , address=145.79.227.51:4411 , piece_id=7JDMY4F74SKL6NSFN2HO36EGUCM5MWVWR5HLXACHBLZXTDRRWI3Q, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:20.781591407 (497h13m20.652674452s) [noise=true , address=145.79.227.46:4603 , piece_id=7L43455SH4JVSXBJYL3WVMRCWFQZ2CEI6DQA7MUXDYSKPPKESY7A, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:20.787415879 (497h13m20.658498924s) [noise=true , address=145.79.227.45:4229 , piece_id=GNXPJKQLV6ITICMBLE4PY2ODOKEJ6WWBUAPA74AO4IBVOVVEFSLA, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:21.029225186 (497h13m20.900308221s) [noise=true , address=45.91.116.4:4937 , piece_id=7HT3WA63RZNVO3AX775OL37N4BJRXZCZ5P6A2A2Z6NPZX6RASSSQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:21.124342636 (497h13m20.995425671s) [noise=true , address=77.90.31.21:4878 , piece_id=RNZ3EK2ZERZY6IQEBAZPHEUDBHNF7BARJF5FVZMLVVMLH5OK7WCQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:21.148939092 (497h13m21.020022127s) [noise=true , address=77.90.31.14:4664 , piece_id=JFEBDNYO2OQRCKCLBKWS43SZCZ7L2LSIF5FP2JS37FFARUEHCESA, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:21.733228030 (497h13m21.604311075s) [noise=true , address=77.90.31.11:4555 , piece_id=ZTSV63NMIHDFU464BAUD3MH3645MJOCGLTDQEKIRPD47EMLLO5EA, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:21.818455067 (497h13m21.689538102s) [noise=true , address=77.90.31.17:4773 , piece_id=P645R4U4WSCWJR5XZSGFGVVF3QHAVSP6SN46CPQGBT6JWJB57ZIQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:21.818474194 (497h13m21.689557239s) [seg_pos={0 0}, upload=436400]: Segment reached optimal threshold of 13 pieces.
Dec 26 08:42:21.818682150 (497h13m21.689765195s) [noise=true , address=77.90.31.10:4517 , piece_id=RD3ARVUTAOQRS3YUQ7WL2LF47CXQFSVDDJSJUAEZAFVGUU4HJ7KQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF
Dec 26 08:42:21.818690926 (497h13m21.689773971s) [noise=true , address=77.90.31.10:4517 , piece_id=RD3ARVUTAOQRS3YUQ7WL2LF47CXQFSVDDJSJUAEZAFVGUU4HJ7KQ, seg_pos={0 0}, upload=436400]: Long tail cancellation due to optimal threshold
Dec 26 08:42:21.818911596 (497h13m21.689994631s) [noise=true , address=145.79.227.40:4051 , piece_id=NLPGNO63SJLYA65HYN2CJBJFS6OTLX7ISULHSM6Q54PWOOTVAVWQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF
Dec 26 08:42:21.818918088 (497h13m21.690001123s) [noise=true , address=145.79.227.40:4051 , piece_id=NLPGNO63SJLYA65HYN2CJBJFS6OTLX7ISULHSM6Q54PWOOTVAVWQ, seg_pos={0 0}, upload=436400]: Long tail cancellation due to optimal threshold
Dec 26 08:42:21.818941363 (497h13m21.690024418s) [noise=true , address=77.90.31.18:4808 , piece_id=INW543LSVQLUQQ4FZZXVTVPV2LSGPKMUVHCSKE3OQ2CQAIEQFD2Q, seg_pos={0 0}, upload=436400]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: expected piece hash; context canceled; EOF
Dec 26 08:42:21.818997390 (497h13m21.690080425s) [noise=true , address=77.90.31.18:4808 , piece_id=INW543LSVQLUQQ4FZZXVTVPV2LSGPKMUVHCSKE3OQ2CQAIEQFD2Q, seg_pos={0 0}, upload=436400]: Long tail cancellation due to optimal threshold
Dec 26 08:42:21.819032707 (497h13m21.690115742s) [seg_pos={0 0}, upload=436400]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 26 08:42:21.819224622 (497h13m21.690307657s) [upload=436400]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 08:42:21.819232357 (497h13m21.690315392s) [upload=436400]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 08:42:22.109308312 (497h13m21.980391357s) [upload=436400]: Done uploading segments...
Dec 26 08:42:22.109331417 (497h13m21.980414452s) [upload=436400]: Upload finished. err: <nil>
Duration: 5.739302715s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [..........X.................................................] duration:
3.476187398s trace:
https://jaeger.dev.storj.io/trace/185fddc4dc40eba9
FAIL (2025-12-26 08:45:39 UTC)
orbiter warning: context deadline exceeded
Duration: 3.476211865s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [............X...............................................] duration:
5.895213406s trace:
https://jaeger.dev.storj.io/trace/4b8b74dd4906ed4
FAIL (2025-12-26 08:45:26 UTC)
orbiter warning: context deadline exceeded
Duration: 5.895241609s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.............................................X..............] duration:
36.236286869s trace:
https://jaeger.dev.storj.io/trace/6820217d73ecf2a
FAIL (2025-12-26 08:45:24 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
ss=45.91.116.13:4174 , piece_id=RLKVQ3QJM4STACAOUTWA7STEIG4PITJRIB4YOI3GNTYMPA76IFHQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.705106031 (497h3m36.576189076s) [noise=true , address=77.90.31.19:4870 , piece_id=ZHJNUA7P36RBWXQXMDLFRIHQPSZNOPGL67AQVDTDEPCRY2R4EMWQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.705493759 (497h3m36.576576804s) [noise=true , address=145.79.227.20:4443 , piece_id=HAFVT75GMRHFRR5E5VD4AOJFHGRUS56MQX3HNHI6FQYYMZT4J4FQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.705630279 (497h3m36.576713334s) [noise=true , address=77.90.31.15:4717 , piece_id=CVL5BLBIJ5DWSL7ZSEJ4I62RENOFPNWWNZCSE62CTMWGSS2XTE2Q, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.705923348 (497h3m36.577006393s) [noise=true , address=77.90.31.12:4607 , piece_id=TPFAMNN2BEYPSYRYOMDMSS77ZRB35HOJYWMJGTJEERRWGV5KQHHA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706055580 (497h3m36.577138615s) [noise=true , address=64.49.13.169:4705 , piece_id=NG53QDRTA5IAV5W64W6MSZVNCWDUDHBB3DP4BF6G7DTF5DAF7ECA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706362764 (497h3m36.577445819s) [noise=true , address=77.90.31.11:4581 , piece_id=NQOGQYJLH7W7CXFLBDW3XUN6Y6YVQEWAFBMLNKNHKTU5VCV4V3GQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706507651 (497h3m36.577590686s) [noise=true , address=145.79.227.44:4226 , piece_id=XFE2F2254TY5CMO7J5O6S3RDKHOHSLQCUGDNHB37VKOXPM63GJEA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706728992 (497h3m36.577812027s) [noise=true , address=45.91.116.4:4942 , piece_id=VDKHG53ATFYJDQYDBXKZ7SN2RNAAZBYGQVGAF34PCG4UYO7XMFGA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706869399 (497h3m36.577952444s) [noise=true , address=77.90.31.16:4750 , piece_id=HIZWD4OHAZ6UKMODPFHABQRQW7ESEZPVMSD45RI5HGXPLHDJLJHQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707077395 (497h3m36.57816044s) [noise=true , address=145.79.227.34:4909 , piece_id=TSBSOK42K3BBLLTUBV6UFWW33FK67FZHJZTKEQFAAQ7ITKY3RAEQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707229524 (497h3m36.578312559s) [noise=true , address=77.90.31.13:4655 , piece_id=B6QS3X24QQRPWA3O5ENXR32XJOJZBZZOVVDMSESJYKU3XBIPGDIQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707443251 (497h3m36.578526296s) [noise=true , address=145.79.227.5:4241 , piece_id=ANDNWFXV5YWOH55WDLOMJXSKZVN45XSIYT2I4QB52DNO2PT3HYKQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707633815 (497h3m36.57871685s) [noise=true , address=64.49.13.154:4930 , piece_id=IHQYIUPLXMAKL4S76Y6NAGDP2ZWFRQBOXXJJENZ6JRGBXWLKDCFQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707813797 (497h3m36.578896832s) [noise=true , address=145.79.227.13:4822 , piece_id=HSHVLD6IYQOG3KQYCZGENM4ZYH7XCKGRAZLEL6N2WELIBI2LNHSA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.708022374 (497h3m36.579105439s) [noise=true , address=145.79.227.30:4775 , piece_id=OG65SPMQI3XHNJHUHKSJ4NZKGAB764OYFE7D5CBJCP3YRAGYANQA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.708682712 (497h3m36.579765747s) [noise=true , address=77.90.31.17:4788 , piece_id=CIK5XZI3QMG7LFEOKBPZGNLDOORNP6IQV3QYSHLIY5XSF3LLXBBQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:38.013251430 (497h3m37.884334475s) [upload=436256]: Next returned nil segment
Dec 26 08:32:38.013277480 (497h3m37.884360525s) [upload=436256]: Waiting for error group managing segments...
Dec 26 08:32:38.345076384 (497h3m38.216159419s) [noise=true , address=64.49.13.154:4930 , piece_id=IHQYIUPLXMAKL4S76Y6NAGDP2ZWFRQBOXXJJENZ6JRGBXWLKDCFQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.593234781 (497h3m38.464317836s) [noise=true , address=145.79.227.20:4443 , piece_id=HAFVT75GMRHFRR5E5VD4AOJFHGRUS56MQX3HNHI6FQYYMZT4J4FQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.593901300 (497h3m38.464984335s) [noise=true , address=145.79.227.5:4241 , piece_id=ANDNWFXV5YWOH55WDLOMJXSKZVN45XSIYT2I4QB52DNO2PT3HYKQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.594508065 (497h3m38.4655911s) [noise=true , address=145.79.227.13:4822 , piece_id=HSHVLD6IYQOG3KQYCZGENM4ZYH7XCKGRAZLEL6N2WELIBI2LNHSA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.636070587 (497h3m38.507153653s) [noise=true , address=145.79.227.34:4909 , piece_id=TSBSOK42K3BBLLTUBV6UFWW33FK67FZHJZTKEQFAAQ7ITKY3RAEQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.750629848 (497h3m38.621712883s) [noise=true , address=77.90.31.11:4581 , piece_id=NQOGQYJLH7W7CXFLBDW3XUN6Y6YVQEWAFBMLNKNHKTU5VCV4V3GQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.814987737 (497h3m38.686070782s) [noise=true , address=45.91.116.4:4942 , piece_id=VDKHG53ATFYJDQYDBXKZ7SN2RNAAZBYGQVGAF34PCG4UYO7XMFGA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.847327759 (497h3m38.718410794s) [noise=true , address=77.90.31.15:4717 , piece_id=CVL5BLBIJ5DWSL7ZSEJ4I62RENOFPNWWNZCSE62CTMWGSS2XTE2Q, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.902089014 (497h3m38.773172049s) [noise=true , address=45.91.116.13:4174 , piece_id=RLKVQ3QJM4STACAOUTWA7STEIG4PITJRIB4YOI3GNTYMPA76IFHQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.912841608 (497h3m38.783924644s) [noise=true , address=77.90.31.12:4607 , piece_id=TPFAMNN2BEYPSYRYOMDMSS77ZRB35HOJYWMJGTJEERRWGV5KQHHA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.070851326 (497h3m38.941934371s) [noise=true , address=77.90.31.16:4750 , piece_id=HIZWD4OHAZ6UKMODPFHABQRQW7ESEZPVMSD45RI5HGXPLHDJLJHQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.469018898 (497h3m39.340101933s) [noise=true , address=145.79.227.30:4775 , piece_id=OG65SPMQI3XHNJHUHKSJ4NZKGAB764OYFE7D5CBJCP3YRAGYANQA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.477948602 (497h3m39.349031637s) [noise=true , address=145.79.227.44:4226 , piece_id=XFE2F2254TY5CMO7J5O6S3RDKHOHSLQCUGDNHB37VKOXPM63GJEA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.534878005 (497h3m39.40596105s) [noise=true , address=77.90.31.17:4788 , piece_id=CIK5XZI3QMG7LFEOKBPZGNLDOORNP6IQV3QYSHLIY5XSF3LLXBBQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.534902853 (497h3m39.405985898s) [seg_pos={0 0}, upload=436256]: Segment reached optimal threshold of 14 pieces.
Dec 26 08:32:39.535131698 (497h3m39.406214743s) [noise=true , address=77.90.31.19:4870 , piece_id=ZHJNUA7P36RBWXQXMDLFRIHQPSZNOPGL67AQVDTDEPCRY2R4EMWQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hNVXEVe8DW8F8meERGWkxAazEZKLKBeprNn6e5C94eEoAMBaR): protocol: expected piece hash; context canceled; EOF
Dec 26 08:32:39.535140075 (497h3m39.40622311s) [noise=true , address=77.90.31.19:4870 , piece_id=ZHJNUA7P36RBWXQXMDLFRIHQPSZNOPGL67AQVDTDEPCRY2R4EMWQ, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold
Dec 26 08:32:39.535367978 (497h3m39.406451013s) [noise=true , address=77.90.31.18:4838 , piece_id=V5LDUMURFU4H43E2A5MTLWBI6ZJHVPYMUILBDV5N7ERELPLTIBAA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1if7vf1cNTBwAhb6owi6tuGLYqSfa3feWhsvGSevwuwykhz2RY): protocol: expected piece hash; context canceled; EOF
Dec 26 08:32:39.535375542 (497h3m39.406458577s) [noise=true , address=77.90.31.18:4838 , piece_id=V5LDUMURFU4H43E2A5MTLWBI6ZJHVPYMUILBDV5N7ERELPLTIBAA, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold
Dec 26 08:32:39.535565575 (497h3m39.40664861s) [noise=true , address=77.90.31.13:4655 , piece_id=B6QS3X24QQRPWA3O5ENXR32XJOJZBZZOVVDMSESJYKU3XBIPGDIQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1K7fQM1NVA1L7ukmDtyWh2DYWypUoj3GbreETZpPL6gXfXmZDD): protocol: expected piece hash; context canceled; EOF
Dec 26 08:32:39.535606041 (497h3m39.406689077s) [noise=true , address=77.90.31.13:4655 , piece_id=B6QS3X24QQRPWA3O5ENXR32XJOJZBZZOVVDMSESJYKU3XBIPGDIQ, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold
Dec 26 08:32:39.535707996 (497h3m39.406791031s) [noise=true , address=64.49.13.169:4705 , piece_id=NG53QDRTA5IAV5W64W6MSZVNCWDUDHBB3DP4BF6G7DTF5DAF7ECA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: read tcp 188.166.225.170:35726->64.49.13.169:4705: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 26 08:32:39.535715059 (497h3m39.406798094s) [noise=true , address=64.49.13.169:4705 , piece_id=NG53QDRTA5IAV5W64W6MSZVNCWDUDHBB3DP4BF6G7DTF5DAF7ECA, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold
Dec 26 08:32:39.535740878 (497h3m39.406823923s) [seg_pos={0 0}, upload=436256]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 26 08:32:39.535861157 (497h3m39.406944202s) [upload=436256]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 08:32:39.535867620 (497h3m39.406950655s) [upload=436256]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 08:32:39.833497059 (497h3m39.704580164s) [upload=436256]: Done uploading segments...
Dec 26 08:32:39.833522858 (497h3m39.704605903s) [upload=436256]: Upload finished. err: <nil>
Duration: 36.236303894s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
21 out of 60 is failed (threshold=3) [.......X..X.XX.X.X..X..X.XXX.....X......X.X...X.XX..X.XXX...] duration:
5.877429742s trace:
https://jaeger.dev.storj.io/trace/3312cc3668ef2c5f
FAIL (2025-12-26 08:45:11 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 26 08:42:18.506798665 (497h13m18.37788171s) [upload=436400]: Starting upload
Dec 26 08:42:18.506881102 (497h13m18.377964137s) [upload=436400]: Done starting upload
Dec 26 08:42:18.506963669 (497h13m18.378046714s) [upload=436400]: Uploading segments...
Dec 26 08:42:18.507170413 (497h13m18.378253478s) [upload=436400]: Got next segment. Inline: false
Dec 26 08:42:18.507247920 (497h13m18.378330965s) [upload=436400]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 26 08:42:18.507255454 (497h13m18.378338499s) [upload=436400]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 26 08:42:18.754283459 (497h13m18.625366504s) [seg_pos={0 0}, upload=436400]: Begin upload segment...
Dec 26 08:42:18.754356328 (497h13m18.625439373s) [seg_pos={0 0}, upload=436400]: Done begin upload segment.
Dec 26 08:42:18.754391745 (497h13m18.62547479s) [noise=true , address=145.79.227.40:4051 , piece_id=NLPGNO63SJLYA65HYN2CJBJFS6OTLX7ISULHSM6Q54PWOOTVAVWQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.754466177 (497h13m18.625549222s) [noise=true , address=77.90.31.17:4773 , piece_id=P645R4U4WSCWJR5XZSGFGVVF3QHAVSP6SN46CPQGBT6JWJB57ZIQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.755567554 (497h13m18.626650599s) [noise=true , address=77.90.31.18:4808 , piece_id=INW543LSVQLUQQ4FZZXVTVPV2LSGPKMUVHCSKE3OQ2CQAIEQFD2Q, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.755910858 (497h13m18.626993893s) [noise=true , address=145.79.227.17:4970 , piece_id=JXWYFTTKYNB6BSPBKWBK4YPF2PNAGWJKXHJAOUFIJVVIKIVNPUJA, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.756229915 (497h13m18.62731296s) [noise=true , address=145.79.227.46:4603 , piece_id=7L43455SH4JVSXBJYL3WVMRCWFQZ2CEI6DQA7MUXDYSKPPKESY7A, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.756613886 (497h13m18.627696922s) [noise=true , address=145.79.227.16:4912 , piece_id=TIPMXPQDX4HVH2NUQ3FUFT6J34SNM5DT2NHDXCZO76MMGPEEEHMQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.757020902 (497h13m18.628103937s) [noise=true , address=145.79.227.38:4977 , piece_id=MYMT3XBOXWU5DZOTPVHSVVXOKNHW7WWVCBECIO37VAYUTFK7XZSQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.757428849 (497h13m18.628511904s) [noise=true , address=77.90.31.11:4555 , piece_id=ZTSV63NMIHDFU464BAUD3MH3645MJOCGLTDQEKIRPD47EMLLO5EA, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.757902380 (497h13m18.628985425s) [noise=true , address=145.79.227.45:4229 , piece_id=GNXPJKQLV6ITICMBLE4PY2ODOKEJ6WWBUAPA74AO4IBVOVVEFSLA, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.757979276 (497h13m18.629062321s) [noise=true , address=77.90.31.21:4878 , piece_id=RNZ3EK2ZERZY6IQEBAZPHEUDBHNF7BARJF5FVZMLVVMLH5OK7WCQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.758365802 (497h13m18.629448837s) [noise=true , address=45.91.116.4:4937 , piece_id=7HT3WA63RZNVO3AX775OL37N4BJRXZCZ5P6A2A2Z6NPZX6RASSSQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.758773228 (497h13m18.629856263s) [noise=true , address=77.90.31.10:4517 , piece_id=RD3ARVUTAOQRS3YUQ7WL2LF47CXQFSVDDJSJUAEZAFVGUU4HJ7KQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.759342462 (497h13m18.630425507s) [noise=true , address=145.79.227.37:4567 , piece_id=J7Z3543ZQ2TIGFFHBO7V7GJKTGOY54Z6WAPYDI7ALYRYDS6NEK2Q, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.760026635 (497h13m18.63110968s) [noise=true , address=145.79.227.48:4301 , piece_id=IS2W2WAK7NOMKFEHY3QK7KS35XPQYC2723Q2ZJ27HTX4FPBQYHDQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.760798124 (497h13m18.631881169s) [noise=true , address=145.79.227.51:4411 , piece_id=7JDMY4F74SKL6NSFN2HO36EGUCM5MWVWR5HLXACHBLZXTDRRWI3Q, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.761677319 (497h13m18.632760364s) [noise=true , address=77.90.31.14:4664 , piece_id=JFEBDNYO2OQRCKCLBKWS43SZCZ7L2LSIF5FP2JS37FFARUEHCESA, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:20.466851608 (497h13m20.337934643s) [upload=436400]: Next returned nil segment
Dec 26 08:42:20.466865273 (497h13m20.337948318s) [upload=436400]: Waiting for error group managing segments...
Dec 26 08:42:20.750402218 (497h13m20.621485253s) [noise=true , address=145.79.227.37:4567 , piece_id=J7Z3543ZQ2TIGFFHBO7V7GJKTGOY54Z6WAPYDI7ALYRYDS6NEK2Q, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:20.763579740 (497h13m20.634662785s) [noise=true , address=145.79.227.48:4301 , piece_id=IS2W2WAK7NOMKFEHY3QK7KS35XPQYC2723Q2ZJ27HTX4FPBQYHDQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:20.767844120 (497h13m20.638927155s) [noise=true , address=145.79.227.38:4977 , piece_id=MYMT3XBOXWU5DZOTPVHSVVXOKNHW7WWVCBECIO37VAYUTFK7XZSQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:20.771941653 (497h13m20.643024698s) [noise=true , address=145.79.227.16:4912 , piece_id=TIPMXPQDX4HVH2NUQ3FUFT6J34SNM5DT2NHDXCZO76MMGPEEEHMQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:20.773887688 (497h13m20.644970723s) [noise=true , address=145.79.227.17:4970 , piece_id=JXWYFTTKYNB6BSPBKWBK4YPF2PNAGWJKXHJAOUFIJVVIKIVNPUJA, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:20.781414641 (497h13m20.652497676s) [noise=true , address=145.79.227.51:4411 , piece_id=7JDMY4F74SKL6NSFN2HO36EGUCM5MWVWR5HLXACHBLZXTDRRWI3Q, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:20.781591407 (497h13m20.652674452s) [noise=true , address=145.79.227.46:4603 , piece_id=7L43455SH4JVSXBJYL3WVMRCWFQZ2CEI6DQA7MUXDYSKPPKESY7A, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:20.787415879 (497h13m20.658498924s) [noise=true , address=145.79.227.45:4229 , piece_id=GNXPJKQLV6ITICMBLE4PY2ODOKEJ6WWBUAPA74AO4IBVOVVEFSLA, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:21.029225186 (497h13m20.900308221s) [noise=true , address=45.91.116.4:4937 , piece_id=7HT3WA63RZNVO3AX775OL37N4BJRXZCZ5P6A2A2Z6NPZX6RASSSQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:21.124342636 (497h13m20.995425671s) [noise=true , address=77.90.31.21:4878 , piece_id=RNZ3EK2ZERZY6IQEBAZPHEUDBHNF7BARJF5FVZMLVVMLH5OK7WCQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:21.148939092 (497h13m21.020022127s) [noise=true , address=77.90.31.14:4664 , piece_id=JFEBDNYO2OQRCKCLBKWS43SZCZ7L2LSIF5FP2JS37FFARUEHCESA, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:21.733228030 (497h13m21.604311075s) [noise=true , address=77.90.31.11:4555 , piece_id=ZTSV63NMIHDFU464BAUD3MH3645MJOCGLTDQEKIRPD47EMLLO5EA, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:21.818455067 (497h13m21.689538102s) [noise=true , address=77.90.31.17:4773 , piece_id=P645R4U4WSCWJR5XZSGFGVVF3QHAVSP6SN46CPQGBT6JWJB57ZIQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:21.818474194 (497h13m21.689557239s) [seg_pos={0 0}, upload=436400]: Segment reached optimal threshold of 13 pieces.
Dec 26 08:42:21.818682150 (497h13m21.689765195s) [noise=true , address=77.90.31.10:4517 , piece_id=RD3ARVUTAOQRS3YUQ7WL2LF47CXQFSVDDJSJUAEZAFVGUU4HJ7KQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF
Dec 26 08:42:21.818690926 (497h13m21.689773971s) [noise=true , address=77.90.31.10:4517 , piece_id=RD3ARVUTAOQRS3YUQ7WL2LF47CXQFSVDDJSJUAEZAFVGUU4HJ7KQ, seg_pos={0 0}, upload=436400]: Long tail cancellation due to optimal threshold
Dec 26 08:42:21.818911596 (497h13m21.689994631s) [noise=true , address=145.79.227.40:4051 , piece_id=NLPGNO63SJLYA65HYN2CJBJFS6OTLX7ISULHSM6Q54PWOOTVAVWQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF
Dec 26 08:42:21.818918088 (497h13m21.690001123s) [noise=true , address=145.79.227.40:4051 , piece_id=NLPGNO63SJLYA65HYN2CJBJFS6OTLX7ISULHSM6Q54PWOOTVAVWQ, seg_pos={0 0}, upload=436400]: Long tail cancellation due to optimal threshold
Dec 26 08:42:21.818941363 (497h13m21.690024418s) [noise=true , address=77.90.31.18:4808 , piece_id=INW543LSVQLUQQ4FZZXVTVPV2LSGPKMUVHCSKE3OQ2CQAIEQFD2Q, seg_pos={0 0}, upload=436400]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: expected piece hash; context canceled; EOF
Dec 26 08:42:21.818997390 (497h13m21.690080425s) [noise=true , address=77.90.31.18:4808 , piece_id=INW543LSVQLUQQ4FZZXVTVPV2LSGPKMUVHCSKE3OQ2CQAIEQFD2Q, seg_pos={0 0}, upload=436400]: Long tail cancellation due to optimal threshold
Dec 26 08:42:21.819032707 (497h13m21.690115742s) [seg_pos={0 0}, upload=436400]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 26 08:42:21.819224622 (497h13m21.690307657s) [upload=436400]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 08:42:21.819232357 (497h13m21.690315392s) [upload=436400]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 08:42:22.109308312 (497h13m21.980391357s) [upload=436400]: Done uploading segments...
Dec 26 08:42:22.109331417 (497h13m21.980414452s) [upload=436400]: Upload finished. err: <nil>
Duration: 5.877462334s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [...........X................................................] duration:
3.549029122s trace:
https://jaeger.dev.storj.io/trace/185fddc4dc40eba9
FAIL (2025-12-26 08:45:11 UTC)
orbiter warning: context deadline exceeded
Duration: 3.549054741s
core
uplink
us-select-1
us1-placement-20
|