Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
21 out of 60 is failed (threshold=6) [.XX.....X..X.X....X.XXXX....X.X.X...X..X.X.X..X...XX.....X..] duration:
5.00059622s trace:
https://jaeger.dev.storj.io/trace/5bac827d8dc35b4e
FAIL (2025-12-28 03:59:40 UTC)
orbiter warning: context deadline exceeded
Duration: 5.000624554s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [............X...X...........................................] duration:
46.842805118s trace:
https://jaeger.dev.storj.io/trace/54422f75ff432c0
FAIL (2025-12-28 03:59:34 UTC)
orbiter warning:
operation: DeleteObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
, piece_id=SPQSUVHIHI32ZMIRE4NHSNJCMWS4NS6ZZPYF6FV7FQH6VADQIV5A, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:35.720215866 (539h54m35.591298911s) [noise=true , address=145.79.227.43:4187 , piece_id=MRILXYFSWVSH35ZCCFZWLEXENWJII6G6LOR2N277GIBMI4PXPPTQ, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:35.720567665 (539h54m35.5916507s) [noise=true , address=145.79.227.33:4896 , piece_id=7NRM7I4IDHW2WGOCG3B6L6HN74ANXWDJCFRPBM4BCHWJELSNNUXA, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:35.720883607 (539h54m35.591966652s) [noise=true , address=145.79.227.40:4062 , piece_id=RCVKQLDSWIWX2FQBTUFPEUXQZLP2C3ZCLV2JWHOV7CMJ7ZBNWZNQ, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:35.721874904 (539h54m35.592957949s) [noise=true , address=64.49.13.140:4513 , piece_id=Y3HW5I2FX2CWSPVUS7CU3JPNN2RHP2ASL757AUYINBKRKCLXAJGQ, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:35.722249658 (539h54m35.593332693s) [noise=true , address=64.49.13.169:4715 , piece_id=FRVNWFP2AE4DXE4RKZRE6WTK43T74BTTYM2U7E4P3B4XZIMC2QMQ, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:35.722786881 (539h54m35.593869926s) [noise=true , address=77.90.31.14:4673 , piece_id=ZIB5VOFL7PJSRFDV3HP5XAKYHOJXRVHCYDPIHKRYPNM5ROAY5DTQ, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:35.723112100 (539h54m35.594195135s) [noise=true , address=77.90.31.17:4779 , piece_id=TLU5ZLZRVAMGQLSE2LLNIUE4WOWJBMFJGFQKC3KOUJ23V3UVFMRA, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:35.723440756 (539h54m35.594523791s) [noise=true , address=145.79.227.37:4569 , piece_id=DUXSQUT36XUF66ZVQ2V7ZWNXA6JZKBDO5RXT3QVT26GE2FTH3QXQ, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:35.723849544 (539h54m35.594932589s) [noise=true , address=145.79.227.27:4723 , piece_id=4DECSGDNZLXNQVPU7QALLPFQYDO7MOG6TIBNWQ5UHNVCIHB4GUYQ, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:35.724241290 (539h54m35.595324335s) [noise=true , address=77.90.31.11:4587 , piece_id=6467VZCLWHJTZ36ENMEOEOU2E4MAUCDL2XHNAOMJTCCTUSN33NFA, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:35.724607808 (539h54m35.595690854s) [noise=true , address=145.79.227.47:4272 , piece_id=FL2U4W2YCV5ANDVFHB7H4TFT3SYTBGNLI3KZDI7MY2K6NTSMRUAA, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:35.724974035 (539h54m35.59605707s) [noise=true , address=145.79.227.18:4040 , piece_id=W55GK3QDHRHNR6YJ7JKVZMIEKFQLZSTXKSQ5OMW3U7XRI44EM6RA, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:35.725302281 (539h54m35.596385316s) [noise=true , address=145.79.227.38:4989 , piece_id=OUINZ6BXOLFCDNCXMX7PPFSKWMJ7IPZWO7DAGLGVV2AB7UHO3JYQ, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:35.726824420 (539h54m35.597907455s) [noise=true , address=145.79.227.26:4669 , piece_id=5D7SGVO274FEZ4JVESTMUYQ3RZ3DJA5DL2QZV5YKMXEB4TM6PHAA, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:37.063163473 (539h54m36.934246518s) [upload=473490]: Next returned nil segment
Dec 28 03:23:37.063176499 (539h54m36.934259534s) [upload=473490]: Waiting for error group managing segments...
Dec 28 03:23:37.416516311 (539h54m37.287599376s) [noise=true , address=64.49.13.140:4513 , piece_id=Y3HW5I2FX2CWSPVUS7CU3JPNN2RHP2ASL757AUYINBKRKCLXAJGQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil>
Dec 28 03:23:37.606629586 (539h54m37.477712622s) [noise=true , address=145.79.227.27:4723 , piece_id=4DECSGDNZLXNQVPU7QALLPFQYDO7MOG6TIBNWQ5UHNVCIHB4GUYQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil>
Dec 28 03:23:37.617730204 (539h54m37.488813249s) [noise=true , address=145.79.227.26:4669 , piece_id=5D7SGVO274FEZ4JVESTMUYQ3RZ3DJA5DL2QZV5YKMXEB4TM6PHAA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil>
Dec 28 03:23:37.632542950 (539h54m37.503625995s) [noise=true , address=145.79.227.38:4989 , piece_id=OUINZ6BXOLFCDNCXMX7PPFSKWMJ7IPZWO7DAGLGVV2AB7UHO3JYQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil>
Dec 28 03:23:37.643987531 (539h54m37.515070566s) [noise=true , address=145.79.227.47:4272 , piece_id=FL2U4W2YCV5ANDVFHB7H4TFT3SYTBGNLI3KZDI7MY2K6NTSMRUAA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil>
Dec 28 03:23:37.652913619 (539h54m37.523996654s) [noise=true , address=145.79.227.18:4040 , piece_id=W55GK3QDHRHNR6YJ7JKVZMIEKFQLZSTXKSQ5OMW3U7XRI44EM6RA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil>
Dec 28 03:23:37.661966357 (539h54m37.533049392s) [noise=true , address=145.79.227.37:4569 , piece_id=DUXSQUT36XUF66ZVQ2V7ZWNXA6JZKBDO5RXT3QVT26GE2FTH3QXQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil>
Dec 28 03:23:37.778627848 (539h54m37.649710893s) [noise=true , address=77.90.31.11:4561 , piece_id=HIUCZRRZOMU4RBYZU5SADJAOS4RVML2I6PILIX2V65RS22EBN42Q, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil>
Dec 28 03:23:37.815974812 (539h54m37.687057857s) [noise=true , address=45.91.116.14:4217 , piece_id=SPQSUVHIHI32ZMIRE4NHSNJCMWS4NS6ZZPYF6FV7FQH6VADQIV5A, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil>
Dec 28 03:23:37.861731780 (539h54m37.732814815s) [noise=true , address=77.90.31.14:4673 , piece_id=ZIB5VOFL7PJSRFDV3HP5XAKYHOJXRVHCYDPIHKRYPNM5ROAY5DTQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil>
Dec 28 03:23:38.093273211 (539h54m37.964356256s) [noise=true , address=77.90.31.11:4587 , piece_id=6467VZCLWHJTZ36ENMEOEOU2E4MAUCDL2XHNAOMJTCCTUSN33NFA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil>
Dec 28 03:23:38.267694395 (539h54m38.13877745s) [noise=true , address=77.90.31.17:4779 , piece_id=TLU5ZLZRVAMGQLSE2LLNIUE4WOWJBMFJGFQKC3KOUJ23V3UVFMRA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil>
Dec 28 03:23:38.540947761 (539h54m38.412030796s) [noise=true , address=145.79.227.40:4062 , piece_id=RCVKQLDSWIWX2FQBTUFPEUXQZLP2C3ZCLV2JWHOV7CMJ7ZBNWZNQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil>
Dec 28 03:23:38.815393527 (539h54m38.686476562s) [noise=true , address=145.79.227.43:4187 , piece_id=MRILXYFSWVSH35ZCCFZWLEXENWJII6G6LOR2N277GIBMI4PXPPTQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil>
Dec 28 03:23:38.815412894 (539h54m38.686495939s) [seg_pos={0 0}, upload=473490]: Segment reached optimal threshold of 14 pieces.
Dec 28 03:23:38.815594951 (539h54m38.686677986s) [noise=true , address=64.49.13.169:4715 , piece_id=FRVNWFP2AE4DXE4RKZRE6WTK43T74BTTYM2U7E4P3B4XZIMC2QMQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:43502->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:43502->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43502->64.49.13.169:4715: use of closed network connection
Dec 28 03:23:38.815601032 (539h54m38.686684077s) [noise=true , address=64.49.13.169:4715 , piece_id=FRVNWFP2AE4DXE4RKZRE6WTK43T74BTTYM2U7E4P3B4XZIMC2QMQ, seg_pos={0 0}, upload=473490]: Long tail cancellation due to optimal threshold
Dec 28 03:23:38.815826983 (539h54m38.686910018s) [noise=true , address=145.79.227.41:4118 , piece_id=C7JJ3NHR5355M64G2EF2WWERQVGWR243RQHDV3GBAIC3A66JDMHQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128w37UHxJB9R49iwmHbATt3bXUb1FMTQJ3Wjka7So7eqKWg4iF): protocol: expected piece hash; context canceled; EOF
Dec 28 03:23:38.815834507 (539h54m38.686917542s) [noise=true , address=145.79.227.41:4118 , piece_id=C7JJ3NHR5355M64G2EF2WWERQVGWR243RQHDV3GBAIC3A66JDMHQ, seg_pos={0 0}, upload=473490]: Long tail cancellation due to optimal threshold
Dec 28 03:23:38.815890433 (539h54m38.686973468s) [noise=true , address=145.79.227.33:4896 , piece_id=7NRM7I4IDHW2WGOCG3B6L6HN74ANXWDJCFRPBM4BCHWJELSNNUXA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ndSgjgfZMVQni5tw1ugdTrhJt4c4NagdvgYTbSbjatYYtfu1o): protocol: expected piece hash; context canceled; EOF
Dec 28 03:23:38.815914619 (539h54m38.686997654s) [noise=true , address=145.79.227.33:4896 , piece_id=7NRM7I4IDHW2WGOCG3B6L6HN74ANXWDJCFRPBM4BCHWJELSNNUXA, seg_pos={0 0}, upload=473490]: Long tail cancellation due to optimal threshold
Dec 28 03:23:38.816105613 (539h54m38.687188648s) [noise=true , address=145.79.227.28:4731 , piece_id=EPW3LUA3EWRCURHZ66NCRCA36CNIW4GYQMFNKPZZTRCB3YBGTKJA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NQzTwVYQa6ZzxAHQm1apkwWnqJstTnAJQyHKhBzC2jTN9kEVK): protocol: write tcp 188.166.225.170:58940->145.79.227.28:4731: use of closed network connection; write tcp 188.166.225.170:58940->145.79.227.28:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58940->145.79.227.28:4731: use of closed network connection
Dec 28 03:23:38.816112426 (539h54m38.687195461s) [noise=true , address=145.79.227.28:4731 , piece_id=EPW3LUA3EWRCURHZ66NCRCA36CNIW4GYQMFNKPZZTRCB3YBGTKJA, seg_pos={0 0}, upload=473490]: Long tail cancellation due to optimal threshold
Dec 28 03:23:38.816128436 (539h54m38.687211481s) [seg_pos={0 0}, upload=473490]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 28 03:23:38.816257582 (539h54m38.687340627s) [upload=473490]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 03:23:38.816265778 (539h54m38.687348813s) [upload=473490]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 03:23:39.111115407 (539h54m38.982198442s) [upload=473490]: Done uploading segments...
Dec 28 03:23:39.111138972 (539h54m38.982222008s) [upload=473490]: Upload finished. err: <nil>
Duration: 46.842833753s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..................................X.........................] duration:
6.979077859s trace:
https://jaeger.dev.storj.io/trace/3b8130f32e60318d
FAIL (2025-12-28 03:59:21 UTC)
orbiter warning: context deadline exceeded
Duration: 6.979102427s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
53 out of 60 is failed (threshold=3) [XXX.XXXXXXXXXXXXXXXXX.XXXXX.XXXXXXXXXXXXXXXXX..XXXX.XXXX.XXX] duration:
30.004693983s trace:
https://jaeger.dev.storj.io/trace/71ac8f6199590809
FAIL (2025-12-28 03:59:16 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: Q5OO2MXUYWJ2HS5R4MR62RLKWY46756BDLOPUKS53KH6KCFKDXKA) context deadline exceeded; piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: 3PWBLV4SJDNEFKV6A76LWESTF5IQEYLANTMNFXOS6GA67GSZIOPA) context deadline exceeded
======LOGS======
28 03:58:47.009997422 (540h29m46.881080457s) [upload=474004]: Starting upload
Dec 28 03:58:47.010051714 (540h29m46.88113475s) [upload=474004]: Done starting upload
Dec 28 03:58:47.010245574 (540h29m46.881328609s) [upload=474004]: Uploading segments...
Dec 28 03:58:47.010295208 (540h29m46.881378253s) [upload=474004]: Got next segment. Inline: false
Dec 28 03:58:47.010332880 (540h29m46.881415925s) [upload=474004]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 28 03:58:47.010338711 (540h29m46.881421746s) [upload=474004]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 28 03:58:47.256395716 (540h29m47.127478751s) [seg_pos={0 0}, upload=474004]: Begin upload segment...
Dec 28 03:58:47.256455149 (540h29m47.127538184s) [seg_pos={0 0}, upload=474004]: Done begin upload segment.
Dec 28 03:58:47.256500897 (540h29m47.127583932s) [noise=true , address=145.79.227.41:4086 , piece_id=LCQHRIBMWKUZCEOAM36N2PZDWQDBJGRMZSPN7ONK7RIU3H3WTQAA, seg_pos={0 0}, upload=474004]: Uploading piece...
Dec 28 03:58:47.256550541 (540h29m47.127633586s) [noise=true , address=45.91.116.8:4047 , piece_id=4VAIXCXAOA2ABTF6BP647DFO26NOWKVV4RLHZZNOFJZ7765ZUAUA, seg_pos={0 0}, upload=474004]: Uploading piece...
Dec 28 03:58:47.257159140 (540h29m47.128242185s) [noise=true , address=145.79.227.4:4187 , piece_id=D4T5A5AZ44EULG7Y4OTJG6OSBDE55Z3MJ7NYMYU4YMPZFAOHBR5A, seg_pos={0 0}, upload=474004]: Uploading piece...
Dec 28 03:58:47.257607313 (540h29m47.128690358s) [noise=true , address=145.79.227.16:4910 , piece_id=QTLQFFZPWNAUVS7Q63IRWRIVXN6H5GQ77LYXNZPJQTB2CERG5UHQ, seg_pos={0 0}, upload=474004]: Uploading piece...
Dec 28 03:58:47.258103368 (540h29m47.129186413s) [noise=true , address=45.91.116.13:4155 , piece_id=36EOJAASWUCAF4PESS6XU7WYP3EMFQERDYSCXY34RJXSOYCJOYOA, seg_pos={0 0}, upload=474004]: Uploading piece...
Dec 28 03:58:47.258561330 (540h29m47.129644375s) [noise=true , address=145.79.227.52:4445 , piece_id=TL5WCXIPUQLY3MKRSU37BDZDV6ZWP3GHUMIYVIQSEPCIXUGU3WYQ, seg_pos={0 0}, upload=474004]: Uploading piece...
Dec 28 03:58:47.259110625 (540h29m47.13019366s) [noise=true , address=145.79.227.50:4373 , piece_id=WPQL2RUN3QL3Q5OXDDLYZKSBCJ6RBGPURJ6J5UT4ST5GLXVO3HPA, seg_pos={0 0}, upload=474004]: Uploading piece...
Dec 28 03:58:47.260237962 (540h29m47.131320997s) [noise=true , address=45.91.116.12:4227 , piece_id=Q5OO2MXUYWJ2HS5R4MR62RLKWY46756BDLOPUKS53KH6KCFKDXKA, seg_pos={0 0}, upload=474004]: Uploading piece...
Dec 28 03:58:47.260593399 (540h29m47.131676434s) [noise=true , address=45.91.116.6:4974 , piece_id=3PWBLV4SJDNEFKV6A76LWESTF5IQEYLANTMNFXOS6GA67GSZIOPA, seg_pos={0 0}, upload=474004]: Uploading piece...
Dec 28 03:58:47.261495126 (540h29m47.132578171s) [noise=true , address=145.79.227.10:4402 , piece_id=UUC7AJ7O572JK3VB6EMA6KWQ3BTGDVAPXKVKNM5JWN4NNTQVTGWQ, seg_pos={0 0}, upload=474004]: Uploading piece...
Dec 28 03:58:47.262153409 (540h29m47.133236454s) [noise=true , address=45.91.116.10:4084 , piece_id=FYE45JLTUIVXA4WBKJ5RHYJEKCWSDR5VSC25CEUZZOGVT4LVTCJA, seg_pos={0 0}, upload=474004]: Uploading piece...
Dec 28 03:58:47.263265486 (540h29m47.134348521s) [noise=true , address=45.91.116.14:4189 , piece_id=2QREOSSCCNR5ECGCROKMTCQTTLK5FZKYHXQF5XJXRCACZT5HFNYQ, seg_pos={0 0}, upload=474004]: Uploading piece...
Dec 28 03:58:47.265073469 (540h29m47.136156544s) [noise=true , address=145.79.227.48:4302 , piece_id=G7TL4UJSU5YBHNOSDTM5EZBA27PL2EBIHYS7XHMJVOQE6K47LN2A, seg_pos={0 0}, upload=474004]: Uploading piece...
Dec 28 03:58:47.267101772 (540h29m47.138184817s) [noise=true , address=145.79.227.15:4874 , piece_id=NHVIAW6WSK7YSYDGM7UE5W6I72345QAQDWSLATXYMZHE725MMN6Q, seg_pos={0 0}, upload=474004]: Uploading piece...
Dec 28 03:58:47.269038189 (540h29m47.140121224s) [noise=true , address=45.91.116.7:4010 , piece_id=IDGL7BUJC7ZBDWSVRBCR5NRUCZNPSNIHANFPQNEFJBSDHJOPHTIQ, seg_pos={0 0}, upload=474004]: Uploading piece...
Dec 28 03:58:47.270344376 (540h29m47.141427411s) [noise=true , address=145.79.227.11:4730 , piece_id=RTH35OSTVLWBH72MFT3RNLKMAZNPUSVYX3KRXTFNQNFWE76IVJZA, seg_pos={0 0}, upload=474004]: Uploading piece...
Dec 28 03:58:48.975532446 (540h29m48.846615491s) [upload=474004]: Next returned nil segment
Dec 28 03:58:48.975548647 (540h29m48.846631682s) [upload=474004]: Waiting for error group managing segments...
Dec 28 03:58:49.264019385 (540h29m49.13510242s) [noise=true , address=145.79.227.52:4445 , piece_id=TL5WCXIPUQLY3MKRSU37BDZDV6ZWP3GHUMIYVIQSEPCIXUGU3WYQ, seg_pos={0 0}, upload=474004]: Done uploading piece. err: <nil>
Dec 28 03:58:49.271566927 (540h29m49.142649962s) [noise=true , address=145.79.227.15:4874 , piece_id=NHVIAW6WSK7YSYDGM7UE5W6I72345QAQDWSLATXYMZHE725MMN6Q, seg_pos={0 0}, upload=474004]: Done uploading piece. err: <nil>
Dec 28 03:58:49.276712625 (540h29m49.14779567s) [noise=true , address=145.79.227.4:4187 , piece_id=D4T5A5AZ44EULG7Y4OTJG6OSBDE55Z3MJ7NYMYU4YMPZFAOHBR5A, seg_pos={0 0}, upload=474004]: Done uploading piece. err: <nil>
Dec 28 03:58:49.289280806 (540h29m49.160363841s) [noise=true , address=145.79.227.10:4402 , piece_id=UUC7AJ7O572JK3VB6EMA6KWQ3BTGDVAPXKVKNM5JWN4NNTQVTGWQ, seg_pos={0 0}, upload=474004]: Done uploading piece. err: <nil>
Dec 28 03:58:49.298391715 (540h29m49.16947475s) [noise=true , address=145.79.227.50:4373 , piece_id=WPQL2RUN3QL3Q5OXDDLYZKSBCJ6RBGPURJ6J5UT4ST5GLXVO3HPA, seg_pos={0 0}, upload=474004]: Done uploading piece. err: <nil>
Dec 28 03:58:49.298716134 (540h29m49.169799169s) [noise=true , address=145.79.227.11:4730 , piece_id=RTH35OSTVLWBH72MFT3RNLKMAZNPUSVYX3KRXTFNQNFWE76IVJZA, seg_pos={0 0}, upload=474004]: Done uploading piece. err: <nil>
Dec 28 03:58:49.309136926 (540h29m49.180219961s) [noise=true , address=145.79.227.41:4086 , piece_id=LCQHRIBMWKUZCEOAM36N2PZDWQDBJGRMZSPN7ONK7RIU3H3WTQAA, seg_pos={0 0}, upload=474004]: Done uploading piece. err: <nil>
Dec 28 03:58:49.403609959 (540h29m49.274692994s) [noise=true , address=145.79.227.16:4910 , piece_id=QTLQFFZPWNAUVS7Q63IRWRIVXN6H5GQ77LYXNZPJQTB2CERG5UHQ, seg_pos={0 0}, upload=474004]: Done uploading piece. err: <nil>
Dec 28 03:58:49.495429599 (540h29m49.366512634s) [noise=true , address=45.91.116.10:4084 , piece_id=FYE45JLTUIVXA4WBKJ5RHYJEKCWSDR5VSC25CEUZZOGVT4LVTCJA, seg_pos={0 0}, upload=474004]: Done uploading piece. err: <nil>
Dec 28 03:58:49.503597933 (540h29m49.374680968s) [noise=true , address=45.91.116.8:4047 , piece_id=4VAIXCXAOA2ABTF6BP647DFO26NOWKVV4RLHZZNOFJZ7765ZUAUA, seg_pos={0 0}, upload=474004]: Done uploading piece. err: <nil>
Dec 28 03:58:49.554707435 (540h29m49.42579048s) [noise=true , address=45.91.116.6:4974 , piece_id=3PWBLV4SJDNEFKV6A76LWESTF5IQEYLANTMNFXOS6GA67GSZIOPA, seg_pos={0 0}, upload=474004]: Done uploading piece. err: <nil>
Dec 28 03:58:49.557462572 (540h29m49.428545617s) [noise=true , address=45.91.116.12:4227 , piece_id=Q5OO2MXUYWJ2HS5R4MR62RLKWY46756BDLOPUKS53KH6KCFKDXKA, seg_pos={0 0}, upload=474004]: Done uploading piece. err: <nil>
Dec 28 03:58:49.639578113 (540h29m49.510661148s) [noise=true , address=45.91.116.7:4010 , piece_id=IDGL7BUJC7ZBDWSVRBCR5NRUCZNPSNIHANFPQNEFJBSDHJOPHTIQ, seg_pos={0 0}, upload=474004]: Done uploading piece. err: <nil>
Dec 28 03:58:49.639621585 (540h29m49.51070462s) [seg_pos={0 0}, upload=474004]: Segment reached optimal threshold of 13 pieces.
Dec 28 03:58:49.639831655 (540h29m49.510914721s) [noise=true , address=45.91.116.14:4189 , piece_id=2QREOSSCCNR5ECGCROKMTCQTTLK5FZKYHXQF5XJXRCACZT5HFNYQ, seg_pos={0 0}, upload=474004]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF
Dec 28 03:58:49.639840613 (540h29m49.510923658s) [noise=true , address=45.91.116.14:4189 , piece_id=2QREOSSCCNR5ECGCROKMTCQTTLK5FZKYHXQF5XJXRCACZT5HFNYQ, seg_pos={0 0}, upload=474004]: Long tail cancellation due to optimal threshold
Dec 28 03:58:49.639997081 (540h29m49.511080116s) [noise=true , address=145.79.227.48:4302 , piece_id=G7TL4UJSU5YBHNOSDTM5EZBA27PL2EBIHYS7XHMJVOQE6K47LN2A, seg_pos={0 0}, upload=474004]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tGpuYdPLiy4ncDjxrZbQfp7Bq79NVBHDmr4vjaL4qMpudrvgR): protocol: write tcp 188.166.225.170:52364->145.79.227.48:4302: use of closed network connection; write tcp 188.166.225.170:52364->145.79.227.48:4302: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52364->145.79.227.48:4302: use of closed network connection
Dec 28 03:58:49.640030855 (540h29m49.51111389s) [noise=true , address=145.79.227.48:4302 , piece_id=G7TL4UJSU5YBHNOSDTM5EZBA27PL2EBIHYS7XHMJVOQE6K47LN2A, seg_pos={0 0}, upload=474004]: Long tail cancellation due to optimal threshold
Dec 28 03:58:49.640136005 (540h29m49.51121904s) [noise=true , address=45.91.116.13:4155 , piece_id=36EOJAASWUCAF4PESS6XU7WYP3EMFQERDYSCXY34RJXSOYCJOYOA, seg_pos={0 0}, upload=474004]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF
Dec 28 03:58:49.640144150 (540h29m49.511227186s) [noise=true , address=45.91.116.13:4155 , piece_id=36EOJAASWUCAF4PESS6XU7WYP3EMFQERDYSCXY34RJXSOYCJOYOA, seg_pos={0 0}, upload=474004]: Long tail cancellation due to optimal threshold
Dec 28 03:58:49.640162776 (540h29m49.511245811s) [seg_pos={0 0}, upload=474004]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 28 03:58:49.640232319 (540h29m49.511315364s) [upload=474004]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 03:58:49.640238951 (540h29m49.511321986s) [upload=474004]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 03:58:49.937970913 (540h29m49.809053949s) [upload=474004]: Done uploading segments...
Dec 28 03:58:49.937994629 (540h29m49.809077674s) [upload=474004]: Upload finished. err: <nil>
Duration: 30.00471905s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...................................X........................] duration:
6.733776745s trace:
https://jaeger.dev.storj.io/trace/3b8130f32e60318d
FAIL (2025-12-28 03:59:02 UTC)
orbiter warning: context deadline exceeded
Duration: 6.733859933s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
21 out of 60 is failed (threshold=6) [..XX.....X..X.X....X.XXXX....X.X.X...X..X.X.X..X...XX.....X.] duration:
4.618931326s trace:
https://jaeger.dev.storj.io/trace/5bac827d8dc35b4e
FAIL (2025-12-28 03:58:54 UTC)
orbiter warning: context deadline exceeded
Duration: 4.61895423s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [....................................X.......................] duration:
9.037259129s trace:
https://jaeger.dev.storj.io/trace/3b8130f32e60318d
FAIL (2025-12-28 03:58:44 UTC)
orbiter warning: context deadline exceeded
Duration: 9.037309766s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [.............X...X..........................................] duration:
34.426258899s trace:
https://jaeger.dev.storj.io/trace/54422f75ff432c0
FAIL (2025-12-28 03:58:31 UTC)
orbiter warning:
operation: DeleteObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
, piece_id=SPQSUVHIHI32ZMIRE4NHSNJCMWS4NS6ZZPYF6FV7FQH6VADQIV5A, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:35.720215866 (539h54m35.591298911s) [noise=true , address=145.79.227.43:4187 , piece_id=MRILXYFSWVSH35ZCCFZWLEXENWJII6G6LOR2N277GIBMI4PXPPTQ, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:35.720567665 (539h54m35.5916507s) [noise=true , address=145.79.227.33:4896 , piece_id=7NRM7I4IDHW2WGOCG3B6L6HN74ANXWDJCFRPBM4BCHWJELSNNUXA, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:35.720883607 (539h54m35.591966652s) [noise=true , address=145.79.227.40:4062 , piece_id=RCVKQLDSWIWX2FQBTUFPEUXQZLP2C3ZCLV2JWHOV7CMJ7ZBNWZNQ, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:35.721874904 (539h54m35.592957949s) [noise=true , address=64.49.13.140:4513 , piece_id=Y3HW5I2FX2CWSPVUS7CU3JPNN2RHP2ASL757AUYINBKRKCLXAJGQ, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:35.722249658 (539h54m35.593332693s) [noise=true , address=64.49.13.169:4715 , piece_id=FRVNWFP2AE4DXE4RKZRE6WTK43T74BTTYM2U7E4P3B4XZIMC2QMQ, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:35.722786881 (539h54m35.593869926s) [noise=true , address=77.90.31.14:4673 , piece_id=ZIB5VOFL7PJSRFDV3HP5XAKYHOJXRVHCYDPIHKRYPNM5ROAY5DTQ, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:35.723112100 (539h54m35.594195135s) [noise=true , address=77.90.31.17:4779 , piece_id=TLU5ZLZRVAMGQLSE2LLNIUE4WOWJBMFJGFQKC3KOUJ23V3UVFMRA, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:35.723440756 (539h54m35.594523791s) [noise=true , address=145.79.227.37:4569 , piece_id=DUXSQUT36XUF66ZVQ2V7ZWNXA6JZKBDO5RXT3QVT26GE2FTH3QXQ, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:35.723849544 (539h54m35.594932589s) [noise=true , address=145.79.227.27:4723 , piece_id=4DECSGDNZLXNQVPU7QALLPFQYDO7MOG6TIBNWQ5UHNVCIHB4GUYQ, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:35.724241290 (539h54m35.595324335s) [noise=true , address=77.90.31.11:4587 , piece_id=6467VZCLWHJTZ36ENMEOEOU2E4MAUCDL2XHNAOMJTCCTUSN33NFA, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:35.724607808 (539h54m35.595690854s) [noise=true , address=145.79.227.47:4272 , piece_id=FL2U4W2YCV5ANDVFHB7H4TFT3SYTBGNLI3KZDI7MY2K6NTSMRUAA, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:35.724974035 (539h54m35.59605707s) [noise=true , address=145.79.227.18:4040 , piece_id=W55GK3QDHRHNR6YJ7JKVZMIEKFQLZSTXKSQ5OMW3U7XRI44EM6RA, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:35.725302281 (539h54m35.596385316s) [noise=true , address=145.79.227.38:4989 , piece_id=OUINZ6BXOLFCDNCXMX7PPFSKWMJ7IPZWO7DAGLGVV2AB7UHO3JYQ, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:35.726824420 (539h54m35.597907455s) [noise=true , address=145.79.227.26:4669 , piece_id=5D7SGVO274FEZ4JVESTMUYQ3RZ3DJA5DL2QZV5YKMXEB4TM6PHAA, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:37.063163473 (539h54m36.934246518s) [upload=473490]: Next returned nil segment
Dec 28 03:23:37.063176499 (539h54m36.934259534s) [upload=473490]: Waiting for error group managing segments...
Dec 28 03:23:37.416516311 (539h54m37.287599376s) [noise=true , address=64.49.13.140:4513 , piece_id=Y3HW5I2FX2CWSPVUS7CU3JPNN2RHP2ASL757AUYINBKRKCLXAJGQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil>
Dec 28 03:23:37.606629586 (539h54m37.477712622s) [noise=true , address=145.79.227.27:4723 , piece_id=4DECSGDNZLXNQVPU7QALLPFQYDO7MOG6TIBNWQ5UHNVCIHB4GUYQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil>
Dec 28 03:23:37.617730204 (539h54m37.488813249s) [noise=true , address=145.79.227.26:4669 , piece_id=5D7SGVO274FEZ4JVESTMUYQ3RZ3DJA5DL2QZV5YKMXEB4TM6PHAA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil>
Dec 28 03:23:37.632542950 (539h54m37.503625995s) [noise=true , address=145.79.227.38:4989 , piece_id=OUINZ6BXOLFCDNCXMX7PPFSKWMJ7IPZWO7DAGLGVV2AB7UHO3JYQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil>
Dec 28 03:23:37.643987531 (539h54m37.515070566s) [noise=true , address=145.79.227.47:4272 , piece_id=FL2U4W2YCV5ANDVFHB7H4TFT3SYTBGNLI3KZDI7MY2K6NTSMRUAA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil>
Dec 28 03:23:37.652913619 (539h54m37.523996654s) [noise=true , address=145.79.227.18:4040 , piece_id=W55GK3QDHRHNR6YJ7JKVZMIEKFQLZSTXKSQ5OMW3U7XRI44EM6RA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil>
Dec 28 03:23:37.661966357 (539h54m37.533049392s) [noise=true , address=145.79.227.37:4569 , piece_id=DUXSQUT36XUF66ZVQ2V7ZWNXA6JZKBDO5RXT3QVT26GE2FTH3QXQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil>
Dec 28 03:23:37.778627848 (539h54m37.649710893s) [noise=true , address=77.90.31.11:4561 , piece_id=HIUCZRRZOMU4RBYZU5SADJAOS4RVML2I6PILIX2V65RS22EBN42Q, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil>
Dec 28 03:23:37.815974812 (539h54m37.687057857s) [noise=true , address=45.91.116.14:4217 , piece_id=SPQSUVHIHI32ZMIRE4NHSNJCMWS4NS6ZZPYF6FV7FQH6VADQIV5A, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil>
Dec 28 03:23:37.861731780 (539h54m37.732814815s) [noise=true , address=77.90.31.14:4673 , piece_id=ZIB5VOFL7PJSRFDV3HP5XAKYHOJXRVHCYDPIHKRYPNM5ROAY5DTQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil>
Dec 28 03:23:38.093273211 (539h54m37.964356256s) [noise=true , address=77.90.31.11:4587 , piece_id=6467VZCLWHJTZ36ENMEOEOU2E4MAUCDL2XHNAOMJTCCTUSN33NFA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil>
Dec 28 03:23:38.267694395 (539h54m38.13877745s) [noise=true , address=77.90.31.17:4779 , piece_id=TLU5ZLZRVAMGQLSE2LLNIUE4WOWJBMFJGFQKC3KOUJ23V3UVFMRA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil>
Dec 28 03:23:38.540947761 (539h54m38.412030796s) [noise=true , address=145.79.227.40:4062 , piece_id=RCVKQLDSWIWX2FQBTUFPEUXQZLP2C3ZCLV2JWHOV7CMJ7ZBNWZNQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil>
Dec 28 03:23:38.815393527 (539h54m38.686476562s) [noise=true , address=145.79.227.43:4187 , piece_id=MRILXYFSWVSH35ZCCFZWLEXENWJII6G6LOR2N277GIBMI4PXPPTQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil>
Dec 28 03:23:38.815412894 (539h54m38.686495939s) [seg_pos={0 0}, upload=473490]: Segment reached optimal threshold of 14 pieces.
Dec 28 03:23:38.815594951 (539h54m38.686677986s) [noise=true , address=64.49.13.169:4715 , piece_id=FRVNWFP2AE4DXE4RKZRE6WTK43T74BTTYM2U7E4P3B4XZIMC2QMQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:43502->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:43502->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43502->64.49.13.169:4715: use of closed network connection
Dec 28 03:23:38.815601032 (539h54m38.686684077s) [noise=true , address=64.49.13.169:4715 , piece_id=FRVNWFP2AE4DXE4RKZRE6WTK43T74BTTYM2U7E4P3B4XZIMC2QMQ, seg_pos={0 0}, upload=473490]: Long tail cancellation due to optimal threshold
Dec 28 03:23:38.815826983 (539h54m38.686910018s) [noise=true , address=145.79.227.41:4118 , piece_id=C7JJ3NHR5355M64G2EF2WWERQVGWR243RQHDV3GBAIC3A66JDMHQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128w37UHxJB9R49iwmHbATt3bXUb1FMTQJ3Wjka7So7eqKWg4iF): protocol: expected piece hash; context canceled; EOF
Dec 28 03:23:38.815834507 (539h54m38.686917542s) [noise=true , address=145.79.227.41:4118 , piece_id=C7JJ3NHR5355M64G2EF2WWERQVGWR243RQHDV3GBAIC3A66JDMHQ, seg_pos={0 0}, upload=473490]: Long tail cancellation due to optimal threshold
Dec 28 03:23:38.815890433 (539h54m38.686973468s) [noise=true , address=145.79.227.33:4896 , piece_id=7NRM7I4IDHW2WGOCG3B6L6HN74ANXWDJCFRPBM4BCHWJELSNNUXA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ndSgjgfZMVQni5tw1ugdTrhJt4c4NagdvgYTbSbjatYYtfu1o): protocol: expected piece hash; context canceled; EOF
Dec 28 03:23:38.815914619 (539h54m38.686997654s) [noise=true , address=145.79.227.33:4896 , piece_id=7NRM7I4IDHW2WGOCG3B6L6HN74ANXWDJCFRPBM4BCHWJELSNNUXA, seg_pos={0 0}, upload=473490]: Long tail cancellation due to optimal threshold
Dec 28 03:23:38.816105613 (539h54m38.687188648s) [noise=true , address=145.79.227.28:4731 , piece_id=EPW3LUA3EWRCURHZ66NCRCA36CNIW4GYQMFNKPZZTRCB3YBGTKJA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NQzTwVYQa6ZzxAHQm1apkwWnqJstTnAJQyHKhBzC2jTN9kEVK): protocol: write tcp 188.166.225.170:58940->145.79.227.28:4731: use of closed network connection; write tcp 188.166.225.170:58940->145.79.227.28:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58940->145.79.227.28:4731: use of closed network connection
Dec 28 03:23:38.816112426 (539h54m38.687195461s) [noise=true , address=145.79.227.28:4731 , piece_id=EPW3LUA3EWRCURHZ66NCRCA36CNIW4GYQMFNKPZZTRCB3YBGTKJA, seg_pos={0 0}, upload=473490]: Long tail cancellation due to optimal threshold
Dec 28 03:23:38.816128436 (539h54m38.687211481s) [seg_pos={0 0}, upload=473490]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 28 03:23:38.816257582 (539h54m38.687340627s) [upload=473490]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 03:23:38.816265778 (539h54m38.687348813s) [upload=473490]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 03:23:39.111115407 (539h54m38.982198442s) [upload=473490]: Done uploading segments...
Dec 28 03:23:39.111138972 (539h54m38.982222008s) [upload=473490]: Upload finished. err: <nil>
Duration: 34.426274388s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
21 out of 60 is failed (threshold=6) [...XX.....X..X.X....X.XXXX....X.X.X...X..X.X.X..X...XX.....X] duration:
5.001375407s trace:
https://jaeger.dev.storj.io/trace/5bac827d8dc35b4e
FAIL (2025-12-28 03:58:07 UTC)
orbiter warning: context deadline exceeded
Duration: 5.001399583s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
53 out of 60 is failed (threshold=3) [XXXX.XXXXXXXXXXXXXXXXX.XXXXX.XXXXXXXXXXXXXXXXX..XXXX.XXXX.XX] duration:
30.002123235s trace:
https://jaeger.dev.storj.io/trace/9ee11d62a91b101
FAIL (2025-12-28 03:58:05 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks, Piece ID: GK7MRPDUMUOUNVK3XBCIFSVFBGYT7F7PU3JRVVFYYV2DM262AWQA) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: P66ALNB3ZPGHNUH7IVYLGKDVGPA57IFA4WQD22VMZBGXJUE4PFZA) context deadline exceeded; piecestore: (Node ID: 1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, Piece ID: ALRRJ6AUHCFJHO4IUBQUIELQHBESNY7DE3L6GYQHPGHYQP7AAZJQ) context deadline exceeded
======LOGS======
Dec 28 03:57:35.982954237 (540h28m35.854037272s) [upload=473987]: Starting upload
Dec 28 03:57:35.983020183 (540h28m35.854103218s) [upload=473987]: Done starting upload
Dec 28 03:57:35.983124621 (540h28m35.854207707s) [upload=473987]: Uploading segments...
Dec 28 03:57:35.983262664 (540h28m35.854345709s) [upload=473987]: Got next segment. Inline: false
Dec 28 03:57:35.983377984 (540h28m35.854461019s) [upload=473987]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 28 03:57:35.983383755 (540h28m35.8544668s) [upload=473987]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 28 03:57:36.230544230 (540h28m36.101627266s) [seg_pos={0 0}, upload=473987]: Begin upload segment...
Dec 28 03:57:36.230644592 (540h28m36.101727627s) [seg_pos={0 0}, upload=473987]: Done begin upload segment.
Dec 28 03:57:36.230685039 (540h28m36.101768084s) [noise=true , address=145.79.227.14:4839 , piece_id=MXLNNB76F6YFJXAX443KIH4CQPZZQ6RZPLPTUWFG3HEBSHXIEG2Q, seg_pos={0 0}, upload=473987]: Uploading piece...
Dec 28 03:57:36.230692012 (540h28m36.101775047s) [noise=true , address=145.79.227.41:4086 , piece_id=GQQ7AYHXNI6NTGBS5ZE5TVKBGOA3VBK4WN2SRNPYECUCHRFGCOFQ, seg_pos={0 0}, upload=473987]: Uploading piece...
Dec 28 03:57:36.230741747 (540h28m36.101824792s) [noise=true , address=145.79.227.52:4445 , piece_id=5TI222U3MLXG6W4TLGRUDFOIWHBB2XSOR3C4FEHXWYO3VZVJOH5Q, seg_pos={0 0}, upload=473987]: Uploading piece...
Dec 28 03:57:36.231966138 (540h28m36.103049173s) [noise=true , address=145.79.227.12:4767 , piece_id=KGPE7UCWPALC7YXPL74HOIO2OUKMIKJJUOO4WCQLK3MUAN4GZDXA, seg_pos={0 0}, upload=473987]: Uploading piece...
Dec 28 03:57:36.232292609 (540h28m36.103375655s) [noise=true , address=145.79.227.47:4265 , piece_id=LVYOL2NFU4XZR4COIDWMN6COLNFDJ2S6KJRV6V62DI3OK333XM7Q, seg_pos={0 0}, upload=473987]: Uploading piece...
Dec 28 03:57:36.232703622 (540h28m36.103786657s) [noise=true , address=145.79.227.17:4970 , piece_id=66YLJCK6Y4PCHHD7HYSVJDAQBZNL2B3MLGMR7UVOMIMFR4CILBXA, seg_pos={0 0}, upload=473987]: Uploading piece...
Dec 28 03:57:36.233111670 (540h28m36.104194715s) [noise=true , address=145.79.227.18:4007 , piece_id=3Q444EQXVP52PGWSIF2WHA5N2X56YJC6GFRMNJJ47LU4TX42IYSA, seg_pos={0 0}, upload=473987]: Uploading piece...
Dec 28 03:57:36.234243164 (540h28m36.105326199s) [noise=true , address=45.91.116.11:4119 , piece_id=JCYXT4OG7JWF2KYTQHONGXSWBPDIQNU5JI4IMJJ75CZ4X47BGHQA, seg_pos={0 0}, upload=473987]: Uploading piece...
Dec 28 03:57:36.234546922 (540h28m36.105629957s) [noise=true , address=77.90.31.11:4556 , piece_id=GK7MRPDUMUOUNVK3XBCIFSVFBGYT7F7PU3JRVVFYYV2DM262AWQA, seg_pos={0 0}, upload=473987]: Uploading piece...
Dec 28 03:57:36.235449311 (540h28m36.106532356s) [noise=true , address=45.91.116.5:4300 , piece_id=DQYVFMUTPHH2IMRZ4IUU576USWVDKTN4TE2G7ZRRB67DKDBS7UCA, seg_pos={0 0}, upload=473987]: Uploading piece...
Dec 28 03:57:36.235671413 (540h28m36.106754448s) [noise=true , address=45.91.116.7:4010 , piece_id=F2CA5CCNGG5ZKPF64UOE2KCJZYZO25IJG3IVYKMRPVDMGLBCW4KQ, seg_pos={0 0}, upload=473987]: Uploading piece...
Dec 28 03:57:36.236459695 (540h28m36.10754274s) [noise=true , address=45.91.116.9:4264 , piece_id=5YREAP3CEFBDEQDV6ZADQY5ECRXHT7A3JO2MLAGWDEPFF7UDMWTQ, seg_pos={0 0}, upload=473987]: Uploading piece...
Dec 28 03:57:36.239109430 (540h28m36.110192465s) [noise=true , address=145.79.227.4:4186 , piece_id=BMGMNAYCGNKNK6JRFWKR2ARI34DZGHXMLTEM77P2EV3UIWOPU43A, seg_pos={0 0}, upload=473987]: Uploading piece...
Dec 28 03:57:36.243481456 (540h28m36.114564521s) [noise=true , address=77.90.31.17:4773 , piece_id=ALRRJ6AUHCFJHO4IUBQUIELQHBESNY7DE3L6GYQHPGHYQP7AAZJQ, seg_pos={0 0}, upload=473987]: Uploading piece...
Dec 28 03:57:36.246051760 (540h28m36.117134805s) [noise=true , address=145.79.227.31:4798 , piece_id=CPARCLNMPS6M4VVLRT3C546PIYGS6WNIKOPXLFW2AXV3A22WCF5A, seg_pos={0 0}, upload=473987]: Uploading piece...
Dec 28 03:57:36.248835621 (540h28m36.119918666s) [noise=true , address=45.91.116.6:4973 , piece_id=P66ALNB3ZPGHNUH7IVYLGKDVGPA57IFA4WQD22VMZBGXJUE4PFZA, seg_pos={0 0}, upload=473987]: Uploading piece...
Dec 28 03:57:37.940970155 (540h28m37.8120532s) [upload=473987]: Next returned nil segment
Dec 28 03:57:37.940994421 (540h28m37.812077466s) [upload=473987]: Waiting for error group managing segments...
Dec 28 03:57:38.230517824 (540h28m38.101600859s) [noise=true , address=145.79.227.41:4086 , piece_id=GQQ7AYHXNI6NTGBS5ZE5TVKBGOA3VBK4WN2SRNPYECUCHRFGCOFQ, seg_pos={0 0}, upload=473987]: Done uploading piece. err: <nil>
Dec 28 03:57:38.234241905 (540h28m38.10532495s) [noise=true , address=145.79.227.14:4839 , piece_id=MXLNNB76F6YFJXAX443KIH4CQPZZQ6RZPLPTUWFG3HEBSHXIEG2Q, seg_pos={0 0}, upload=473987]: Done uploading piece. err: <nil>
Dec 28 03:57:38.242140816 (540h28m38.113223861s) [noise=true , address=145.79.227.52:4445 , piece_id=5TI222U3MLXG6W4TLGRUDFOIWHBB2XSOR3C4FEHXWYO3VZVJOH5Q, seg_pos={0 0}, upload=473987]: Done uploading piece. err: <nil>
Dec 28 03:57:38.253681661 (540h28m38.124764696s) [noise=true , address=145.79.227.4:4186 , piece_id=BMGMNAYCGNKNK6JRFWKR2ARI34DZGHXMLTEM77P2EV3UIWOPU43A, seg_pos={0 0}, upload=473987]: Done uploading piece. err: <nil>
Dec 28 03:57:38.257004269 (540h28m38.128087314s) [noise=true , address=145.79.227.31:4798 , piece_id=CPARCLNMPS6M4VVLRT3C546PIYGS6WNIKOPXLFW2AXV3A22WCF5A, seg_pos={0 0}, upload=473987]: Done uploading piece. err: <nil>
Dec 28 03:57:38.257969756 (540h28m38.129052791s) [noise=true , address=145.79.227.18:4007 , piece_id=3Q444EQXVP52PGWSIF2WHA5N2X56YJC6GFRMNJJ47LU4TX42IYSA, seg_pos={0 0}, upload=473987]: Done uploading piece. err: <nil>
Dec 28 03:57:38.264077828 (540h28m38.135160863s) [noise=true , address=145.79.227.47:4265 , piece_id=LVYOL2NFU4XZR4COIDWMN6COLNFDJ2S6KJRV6V62DI3OK333XM7Q, seg_pos={0 0}, upload=473987]: Done uploading piece. err: <nil>
Dec 28 03:57:38.264686516 (540h28m38.135769561s) [noise=true , address=145.79.227.12:4767 , piece_id=KGPE7UCWPALC7YXPL74HOIO2OUKMIKJJUOO4WCQLK3MUAN4GZDXA, seg_pos={0 0}, upload=473987]: Done uploading piece. err: <nil>
Dec 28 03:57:38.272562815 (540h28m38.14364586s) [noise=true , address=145.79.227.17:4970 , piece_id=66YLJCK6Y4PCHHD7HYSVJDAQBZNL2B3MLGMR7UVOMIMFR4CILBXA, seg_pos={0 0}, upload=473987]: Done uploading piece. err: <nil>
Dec 28 03:57:38.479987487 (540h28m38.351070532s) [noise=true , address=45.91.116.6:4973 , piece_id=P66ALNB3ZPGHNUH7IVYLGKDVGPA57IFA4WQD22VMZBGXJUE4PFZA, seg_pos={0 0}, upload=473987]: Done uploading piece. err: <nil>
Dec 28 03:57:38.505328009 (540h28m38.376411044s) [noise=true , address=77.90.31.17:4773 , piece_id=ALRRJ6AUHCFJHO4IUBQUIELQHBESNY7DE3L6GYQHPGHYQP7AAZJQ, seg_pos={0 0}, upload=473987]: Done uploading piece. err: <nil>
Dec 28 03:57:38.553505147 (540h28m38.424588192s) [noise=true , address=77.90.31.11:4556 , piece_id=GK7MRPDUMUOUNVK3XBCIFSVFBGYT7F7PU3JRVVFYYV2DM262AWQA, seg_pos={0 0}, upload=473987]: Done uploading piece. err: <nil>
Dec 28 03:57:38.603355080 (540h28m38.474438115s) [noise=true , address=45.91.116.5:4300 , piece_id=DQYVFMUTPHH2IMRZ4IUU576USWVDKTN4TE2G7ZRRB67DKDBS7UCA, seg_pos={0 0}, upload=473987]: Done uploading piece. err: <nil>
Dec 28 03:57:38.603390818 (540h28m38.474473853s) [seg_pos={0 0}, upload=473987]: Segment reached optimal threshold of 13 pieces.
Dec 28 03:57:38.603609153 (540h28m38.474692218s) [noise=true , address=45.91.116.9:4264 , piece_id=5YREAP3CEFBDEQDV6ZADQY5ECRXHT7A3JO2MLAGWDEPFF7UDMWTQ, seg_pos={0 0}, upload=473987]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: write tcp 188.166.225.170:54014->45.91.116.9:4264: use of closed network connection; write tcp 188.166.225.170:54014->45.91.116.9:4264: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54014->45.91.116.9:4264: use of closed network connection
Dec 28 03:57:38.603620354 (540h28m38.474703399s) [noise=true , address=45.91.116.9:4264 , piece_id=5YREAP3CEFBDEQDV6ZADQY5ECRXHT7A3JO2MLAGWDEPFF7UDMWTQ, seg_pos={0 0}, upload=473987]: Long tail cancellation due to optimal threshold
Dec 28 03:57:38.603686631 (540h28m38.474769676s) [noise=true , address=45.91.116.7:4010 , piece_id=F2CA5CCNGG5ZKPF64UOE2KCJZYZO25IJG3IVYKMRPVDMGLBCW4KQ, seg_pos={0 0}, upload=473987]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF
Dec 28 03:57:38.603696530 (540h28m38.474779575s) [noise=true , address=45.91.116.7:4010 , piece_id=F2CA5CCNGG5ZKPF64UOE2KCJZYZO25IJG3IVYKMRPVDMGLBCW4KQ, seg_pos={0 0}, upload=473987]: Long tail cancellation due to optimal threshold
Dec 28 03:57:38.603919013 (540h28m38.475002079s) [noise=true , address=45.91.116.11:4119 , piece_id=JCYXT4OG7JWF2KYTQHONGXSWBPDIQNU5JI4IMJJ75CZ4X47BGHQA, seg_pos={0 0}, upload=473987]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF
Dec 28 03:57:38.603928331 (540h28m38.475011366s) [noise=true , address=45.91.116.11:4119 , piece_id=JCYXT4OG7JWF2KYTQHONGXSWBPDIQNU5JI4IMJJ75CZ4X47BGHQA, seg_pos={0 0}, upload=473987]: Long tail cancellation due to optimal threshold
Dec 28 03:57:38.603949872 (540h28m38.475032917s) [seg_pos={0 0}, upload=473987]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 28 03:57:38.604035174 (540h28m38.475118219s) [upload=473987]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 03:57:38.604041768 (540h28m38.475124823s) [upload=473987]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 03:57:38.895865980 (540h28m38.766949025s) [upload=473987]: Done uploading segments...
Dec 28 03:57:38.895883594 (540h28m38.766966629s) [upload=473987]: Upload finished. err: <nil>
Duration: 30.002143053s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.....................................X......................] duration:
11.664938013s trace:
https://jaeger.dev.storj.io/trace/3b8130f32e60318d
FAIL (2025-12-28 03:57:51 UTC)
orbiter warning: context deadline exceeded
Duration: 11.664961256s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [..............X...X.........................................] duration:
43.398404307s trace:
https://jaeger.dev.storj.io/trace/54422f75ff432c0
FAIL (2025-12-28 03:57:31 UTC)
orbiter warning:
operation: DeleteObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
, piece_id=SPQSUVHIHI32ZMIRE4NHSNJCMWS4NS6ZZPYF6FV7FQH6VADQIV5A, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:35.720215866 (539h54m35.591298911s) [noise=true , address=145.79.227.43:4187 , piece_id=MRILXYFSWVSH35ZCCFZWLEXENWJII6G6LOR2N277GIBMI4PXPPTQ, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:35.720567665 (539h54m35.5916507s) [noise=true , address=145.79.227.33:4896 , piece_id=7NRM7I4IDHW2WGOCG3B6L6HN74ANXWDJCFRPBM4BCHWJELSNNUXA, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:35.720883607 (539h54m35.591966652s) [noise=true , address=145.79.227.40:4062 , piece_id=RCVKQLDSWIWX2FQBTUFPEUXQZLP2C3ZCLV2JWHOV7CMJ7ZBNWZNQ, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:35.721874904 (539h54m35.592957949s) [noise=true , address=64.49.13.140:4513 , piece_id=Y3HW5I2FX2CWSPVUS7CU3JPNN2RHP2ASL757AUYINBKRKCLXAJGQ, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:35.722249658 (539h54m35.593332693s) [noise=true , address=64.49.13.169:4715 , piece_id=FRVNWFP2AE4DXE4RKZRE6WTK43T74BTTYM2U7E4P3B4XZIMC2QMQ, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:35.722786881 (539h54m35.593869926s) [noise=true , address=77.90.31.14:4673 , piece_id=ZIB5VOFL7PJSRFDV3HP5XAKYHOJXRVHCYDPIHKRYPNM5ROAY5DTQ, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:35.723112100 (539h54m35.594195135s) [noise=true , address=77.90.31.17:4779 , piece_id=TLU5ZLZRVAMGQLSE2LLNIUE4WOWJBMFJGFQKC3KOUJ23V3UVFMRA, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:35.723440756 (539h54m35.594523791s) [noise=true , address=145.79.227.37:4569 , piece_id=DUXSQUT36XUF66ZVQ2V7ZWNXA6JZKBDO5RXT3QVT26GE2FTH3QXQ, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:35.723849544 (539h54m35.594932589s) [noise=true , address=145.79.227.27:4723 , piece_id=4DECSGDNZLXNQVPU7QALLPFQYDO7MOG6TIBNWQ5UHNVCIHB4GUYQ, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:35.724241290 (539h54m35.595324335s) [noise=true , address=77.90.31.11:4587 , piece_id=6467VZCLWHJTZ36ENMEOEOU2E4MAUCDL2XHNAOMJTCCTUSN33NFA, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:35.724607808 (539h54m35.595690854s) [noise=true , address=145.79.227.47:4272 , piece_id=FL2U4W2YCV5ANDVFHB7H4TFT3SYTBGNLI3KZDI7MY2K6NTSMRUAA, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:35.724974035 (539h54m35.59605707s) [noise=true , address=145.79.227.18:4040 , piece_id=W55GK3QDHRHNR6YJ7JKVZMIEKFQLZSTXKSQ5OMW3U7XRI44EM6RA, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:35.725302281 (539h54m35.596385316s) [noise=true , address=145.79.227.38:4989 , piece_id=OUINZ6BXOLFCDNCXMX7PPFSKWMJ7IPZWO7DAGLGVV2AB7UHO3JYQ, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:35.726824420 (539h54m35.597907455s) [noise=true , address=145.79.227.26:4669 , piece_id=5D7SGVO274FEZ4JVESTMUYQ3RZ3DJA5DL2QZV5YKMXEB4TM6PHAA, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:37.063163473 (539h54m36.934246518s) [upload=473490]: Next returned nil segment
Dec 28 03:23:37.063176499 (539h54m36.934259534s) [upload=473490]: Waiting for error group managing segments...
Dec 28 03:23:37.416516311 (539h54m37.287599376s) [noise=true , address=64.49.13.140:4513 , piece_id=Y3HW5I2FX2CWSPVUS7CU3JPNN2RHP2ASL757AUYINBKRKCLXAJGQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil>
Dec 28 03:23:37.606629586 (539h54m37.477712622s) [noise=true , address=145.79.227.27:4723 , piece_id=4DECSGDNZLXNQVPU7QALLPFQYDO7MOG6TIBNWQ5UHNVCIHB4GUYQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil>
Dec 28 03:23:37.617730204 (539h54m37.488813249s) [noise=true , address=145.79.227.26:4669 , piece_id=5D7SGVO274FEZ4JVESTMUYQ3RZ3DJA5DL2QZV5YKMXEB4TM6PHAA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil>
Dec 28 03:23:37.632542950 (539h54m37.503625995s) [noise=true , address=145.79.227.38:4989 , piece_id=OUINZ6BXOLFCDNCXMX7PPFSKWMJ7IPZWO7DAGLGVV2AB7UHO3JYQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil>
Dec 28 03:23:37.643987531 (539h54m37.515070566s) [noise=true , address=145.79.227.47:4272 , piece_id=FL2U4W2YCV5ANDVFHB7H4TFT3SYTBGNLI3KZDI7MY2K6NTSMRUAA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil>
Dec 28 03:23:37.652913619 (539h54m37.523996654s) [noise=true , address=145.79.227.18:4040 , piece_id=W55GK3QDHRHNR6YJ7JKVZMIEKFQLZSTXKSQ5OMW3U7XRI44EM6RA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil>
Dec 28 03:23:37.661966357 (539h54m37.533049392s) [noise=true , address=145.79.227.37:4569 , piece_id=DUXSQUT36XUF66ZVQ2V7ZWNXA6JZKBDO5RXT3QVT26GE2FTH3QXQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil>
Dec 28 03:23:37.778627848 (539h54m37.649710893s) [noise=true , address=77.90.31.11:4561 , piece_id=HIUCZRRZOMU4RBYZU5SADJAOS4RVML2I6PILIX2V65RS22EBN42Q, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil>
Dec 28 03:23:37.815974812 (539h54m37.687057857s) [noise=true , address=45.91.116.14:4217 , piece_id=SPQSUVHIHI32ZMIRE4NHSNJCMWS4NS6ZZPYF6FV7FQH6VADQIV5A, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil>
Dec 28 03:23:37.861731780 (539h54m37.732814815s) [noise=true , address=77.90.31.14:4673 , piece_id=ZIB5VOFL7PJSRFDV3HP5XAKYHOJXRVHCYDPIHKRYPNM5ROAY5DTQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil>
Dec 28 03:23:38.093273211 (539h54m37.964356256s) [noise=true , address=77.90.31.11:4587 , piece_id=6467VZCLWHJTZ36ENMEOEOU2E4MAUCDL2XHNAOMJTCCTUSN33NFA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil>
Dec 28 03:23:38.267694395 (539h54m38.13877745s) [noise=true , address=77.90.31.17:4779 , piece_id=TLU5ZLZRVAMGQLSE2LLNIUE4WOWJBMFJGFQKC3KOUJ23V3UVFMRA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil>
Dec 28 03:23:38.540947761 (539h54m38.412030796s) [noise=true , address=145.79.227.40:4062 , piece_id=RCVKQLDSWIWX2FQBTUFPEUXQZLP2C3ZCLV2JWHOV7CMJ7ZBNWZNQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil>
Dec 28 03:23:38.815393527 (539h54m38.686476562s) [noise=true , address=145.79.227.43:4187 , piece_id=MRILXYFSWVSH35ZCCFZWLEXENWJII6G6LOR2N277GIBMI4PXPPTQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil>
Dec 28 03:23:38.815412894 (539h54m38.686495939s) [seg_pos={0 0}, upload=473490]: Segment reached optimal threshold of 14 pieces.
Dec 28 03:23:38.815594951 (539h54m38.686677986s) [noise=true , address=64.49.13.169:4715 , piece_id=FRVNWFP2AE4DXE4RKZRE6WTK43T74BTTYM2U7E4P3B4XZIMC2QMQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:43502->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:43502->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43502->64.49.13.169:4715: use of closed network connection
Dec 28 03:23:38.815601032 (539h54m38.686684077s) [noise=true , address=64.49.13.169:4715 , piece_id=FRVNWFP2AE4DXE4RKZRE6WTK43T74BTTYM2U7E4P3B4XZIMC2QMQ, seg_pos={0 0}, upload=473490]: Long tail cancellation due to optimal threshold
Dec 28 03:23:38.815826983 (539h54m38.686910018s) [noise=true , address=145.79.227.41:4118 , piece_id=C7JJ3NHR5355M64G2EF2WWERQVGWR243RQHDV3GBAIC3A66JDMHQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128w37UHxJB9R49iwmHbATt3bXUb1FMTQJ3Wjka7So7eqKWg4iF): protocol: expected piece hash; context canceled; EOF
Dec 28 03:23:38.815834507 (539h54m38.686917542s) [noise=true , address=145.79.227.41:4118 , piece_id=C7JJ3NHR5355M64G2EF2WWERQVGWR243RQHDV3GBAIC3A66JDMHQ, seg_pos={0 0}, upload=473490]: Long tail cancellation due to optimal threshold
Dec 28 03:23:38.815890433 (539h54m38.686973468s) [noise=true , address=145.79.227.33:4896 , piece_id=7NRM7I4IDHW2WGOCG3B6L6HN74ANXWDJCFRPBM4BCHWJELSNNUXA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ndSgjgfZMVQni5tw1ugdTrhJt4c4NagdvgYTbSbjatYYtfu1o): protocol: expected piece hash; context canceled; EOF
Dec 28 03:23:38.815914619 (539h54m38.686997654s) [noise=true , address=145.79.227.33:4896 , piece_id=7NRM7I4IDHW2WGOCG3B6L6HN74ANXWDJCFRPBM4BCHWJELSNNUXA, seg_pos={0 0}, upload=473490]: Long tail cancellation due to optimal threshold
Dec 28 03:23:38.816105613 (539h54m38.687188648s) [noise=true , address=145.79.227.28:4731 , piece_id=EPW3LUA3EWRCURHZ66NCRCA36CNIW4GYQMFNKPZZTRCB3YBGTKJA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NQzTwVYQa6ZzxAHQm1apkwWnqJstTnAJQyHKhBzC2jTN9kEVK): protocol: write tcp 188.166.225.170:58940->145.79.227.28:4731: use of closed network connection; write tcp 188.166.225.170:58940->145.79.227.28:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58940->145.79.227.28:4731: use of closed network connection
Dec 28 03:23:38.816112426 (539h54m38.687195461s) [noise=true , address=145.79.227.28:4731 , piece_id=EPW3LUA3EWRCURHZ66NCRCA36CNIW4GYQMFNKPZZTRCB3YBGTKJA, seg_pos={0 0}, upload=473490]: Long tail cancellation due to optimal threshold
Dec 28 03:23:38.816128436 (539h54m38.687211481s) [seg_pos={0 0}, upload=473490]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 28 03:23:38.816257582 (539h54m38.687340627s) [upload=473490]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 03:23:38.816265778 (539h54m38.687348813s) [upload=473490]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 03:23:39.111115407 (539h54m38.982198442s) [upload=473490]: Done uploading segments...
Dec 28 03:23:39.111138972 (539h54m38.982222008s) [upload=473490]: Upload finished. err: <nil>
Duration: 43.398419696s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
53 out of 60 is failed (threshold=3) [XXXXX.XXXXXXXXXXXXXXXXX.XXXXX.XXXXXXXXXXXXXXXXX..XXXX.XXXX.X] duration:
30.003746604s trace:
https://jaeger.dev.storj.io/trace/6fcc0d5cc6cb096b
FAIL (2025-12-28 03:57:04 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: BVVVHPL2V3AJHI3Q6JL5WTLHRNNRJFWDGCEOWQ66YS4GXIFZUDSQ) context deadline exceeded; piecestore: (Node ID: 128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m, Piece ID: J2SKQZEFPXF7ZUWQ2I55WR2NHE5AACWZASV2L3XCHUZ7KLEZ5WKQ) context deadline exceeded; piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: IVPUHKPLQQEHRLYOEAAOQ73HAYV5AGQH2EMRTZZUK3UESBPIXPQQ) context deadline exceeded; piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: TIC7K6VF3UGW32ARXZ6ODCGZB3GTQIKI7KPPFQVSCXC3RKHNQABQ) context deadline exceeded
======LOGS======
8 03:56:34.904439189 (540h27m34.775522234s) [upload=473975]: Starting upload
Dec 28 03:56:34.904498653 (540h27m34.775581708s) [upload=473975]: Done starting upload
Dec 28 03:56:34.904533719 (540h27m34.775616754s) [upload=473975]: Uploading segments...
Dec 28 03:56:34.904763837 (540h27m34.775846882s) [upload=473975]: Got next segment. Inline: false
Dec 28 03:56:34.904807290 (540h27m34.775890325s) [upload=473975]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 28 03:56:34.904812640 (540h27m34.775895675s) [upload=473975]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 28 03:56:35.155106423 (540h27m35.026189478s) [seg_pos={0 0}, upload=473975]: Begin upload segment...
Dec 28 03:56:35.155167408 (540h27m35.026250453s) [seg_pos={0 0}, upload=473975]: Done begin upload segment.
Dec 28 03:56:35.155199670 (540h27m35.026282715s) [noise=true , address=77.90.31.14:4663 , piece_id=TIC7K6VF3UGW32ARXZ6ODCGZB3GTQIKI7KPPFQVSCXC3RKHNQABQ, seg_pos={0 0}, upload=473975]: Uploading piece...
Dec 28 03:56:35.155211994 (540h27m35.026295029s) [noise=true , address=145.79.227.10:4402 , piece_id=FAXKFA23Z6KXWGIK35FPRIPURGP4PZPMUCY5GTNM6XWTI2N2YVCQ, seg_pos={0 0}, upload=473975]: Uploading piece...
Dec 28 03:56:35.155284642 (540h27m35.026367697s) [noise=true , address=77.90.31.15:4700 , piece_id=XJR5JONAZUEEALLJD3GTAO54OXXC5BEZRHARWZE2QD542TM7NZYQ, seg_pos={0 0}, upload=473975]: Uploading piece...
Dec 28 03:56:35.156150551 (540h27m35.027233596s) [noise=true , address=145.79.227.51:4410 , piece_id=ORBNCRG6JZK3ATOEUGQZBCYOQMKOX4AKK6ZJGKPXPEGVRTGRLHNA, seg_pos={0 0}, upload=473975]: Uploading piece...
Dec 28 03:56:35.156652036 (540h27m35.027735081s) [noise=true , address=145.79.227.9:4365 , piece_id=A2ABGJ4CW3CQMQPRWN57WUIZCPCJ3W2K4FNIE3M2B5LORVNTBYMQ, seg_pos={0 0}, upload=473975]: Uploading piece...
Dec 28 03:56:35.157051546 (540h27m35.028134581s) [noise=true , address=45.91.116.7:4011 , piece_id=WDQTIVDOK6QHBE4XEUF2LYQR4ZVU6QBNPJTTIIC3RN2WUXCB34DA, seg_pos={0 0}, upload=473975]: Uploading piece...
Dec 28 03:56:35.157439756 (540h27m35.028522801s) [noise=true , address=145.79.227.42:4122 , piece_id=UJRZKDEHR7KNFU7ZLU2HRV4OI56TY4HTXNEUUKRI3OSHOGTFIBGA, seg_pos={0 0}, upload=473975]: Uploading piece...
Dec 28 03:56:35.157906855 (540h27m35.0289899s) [noise=true , address=145.79.227.41:4087 , piece_id=ZFNLKOERYVBTMBTQHUFFMEPTHUKLKQN3MMNGMPYF2BZH6YX7CU3Q, seg_pos={0 0}, upload=473975]: Uploading piece...
Dec 28 03:56:35.158365117 (540h27m35.029448162s) [noise=true , address=77.90.31.11:4556 , piece_id=HAXLB4XPW3PVBIDQE4G736NCQ5INYDAYNDSM6R7YNXR5ZNSHAK5A, seg_pos={0 0}, upload=473975]: Uploading piece...
Dec 28 03:56:35.158927017 (540h27m35.030010052s) [noise=true , address=145.79.227.37:4566 , piece_id=IUFTERTW4LZWV4QUUP3NOO4AH7MTAQRYIH3LQFRTLQ7GX5BYVIHQ, seg_pos={0 0}, upload=473975]: Uploading piece...
Dec 28 03:56:35.160419640 (540h27m35.031502675s) [noise=true , address=145.79.227.14:4840 , piece_id=GQB554ICYLZNOKKA3QOHSNY4D5S66GYHSM7U2BDI2GZ77RQWS5KQ, seg_pos={0 0}, upload=473975]: Uploading piece...
Dec 28 03:56:35.160433306 (540h27m35.031516351s) [noise=true , address=45.91.116.14:4191 , piece_id=BVVVHPL2V3AJHI3Q6JL5WTLHRNNRJFWDGCEOWQ66YS4GXIFZUDSQ, seg_pos={0 0}, upload=473975]: Uploading piece...
Dec 28 03:56:35.160467842 (540h27m35.031550887s) [noise=true , address=145.79.227.17:4972 , piece_id=3Q6GXIAFGLQF2IKOBCMVNTJV2NSMZ6Y7C3G5A57XSXGVCFPOMNHQ, seg_pos={0 0}, upload=473975]: Uploading piece...
Dec 28 03:56:35.162045125 (540h27m35.03312827s) [noise=true , address=45.91.116.5:4298 , piece_id=IVPUHKPLQQEHRLYOEAAOQ73HAYV5AGQH2EMRTZZUK3UESBPIXPQQ, seg_pos={0 0}, upload=473975]: Uploading piece...
Dec 28 03:56:35.162200540 (540h27m35.033283586s) [noise=true , address=145.79.227.7:4293 , piece_id=HLWHZ2KER3BFELDLOVHAPMUKPRCYZAPJDCITJJBGSPRRKZPPDSKA, seg_pos={0 0}, upload=473975]: Uploading piece...
Dec 28 03:56:35.164035455 (540h27m35.03511849s) [noise=true , address=45.91.116.9:4264 , piece_id=J2SKQZEFPXF7ZUWQ2I55WR2NHE5AACWZASV2L3XCHUZ7KLEZ5WKQ, seg_pos={0 0}, upload=473975]: Uploading piece...
Dec 28 03:56:36.782826560 (540h27m36.653909605s) [upload=473975]: Next returned nil segment
Dec 28 03:56:36.782839876 (540h27m36.653922911s) [upload=473975]: Waiting for error group managing segments...
Dec 28 03:56:37.168576454 (540h27m37.039659499s) [noise=true , address=145.79.227.42:4122 , piece_id=UJRZKDEHR7KNFU7ZLU2HRV4OI56TY4HTXNEUUKRI3OSHOGTFIBGA, seg_pos={0 0}, upload=473975]: Done uploading piece. err: <nil>
Dec 28 03:56:37.170542908 (540h27m37.041625943s) [noise=true , address=145.79.227.10:4402 , piece_id=FAXKFA23Z6KXWGIK35FPRIPURGP4PZPMUCY5GTNM6XWTI2N2YVCQ, seg_pos={0 0}, upload=473975]: Done uploading piece. err: <nil>
Dec 28 03:56:37.179272421 (540h27m37.050355456s) [noise=true , address=145.79.227.37:4566 , piece_id=IUFTERTW4LZWV4QUUP3NOO4AH7MTAQRYIH3LQFRTLQ7GX5BYVIHQ, seg_pos={0 0}, upload=473975]: Done uploading piece. err: <nil>
Dec 28 03:56:37.186878204 (540h27m37.057961239s) [noise=true , address=145.79.227.51:4410 , piece_id=ORBNCRG6JZK3ATOEUGQZBCYOQMKOX4AKK6ZJGKPXPEGVRTGRLHNA, seg_pos={0 0}, upload=473975]: Done uploading piece. err: <nil>
Dec 28 03:56:37.192339413 (540h27m37.063422448s) [noise=true , address=145.79.227.9:4365 , piece_id=A2ABGJ4CW3CQMQPRWN57WUIZCPCJ3W2K4FNIE3M2B5LORVNTBYMQ, seg_pos={0 0}, upload=473975]: Done uploading piece. err: <nil>
Dec 28 03:56:37.208703954 (540h27m37.079786989s) [noise=true , address=145.79.227.7:4293 , piece_id=HLWHZ2KER3BFELDLOVHAPMUKPRCYZAPJDCITJJBGSPRRKZPPDSKA, seg_pos={0 0}, upload=473975]: Done uploading piece. err: <nil>
Dec 28 03:56:37.213431545 (540h27m37.08451458s) [noise=true , address=145.79.227.17:4972 , piece_id=3Q6GXIAFGLQF2IKOBCMVNTJV2NSMZ6Y7C3G5A57XSXGVCFPOMNHQ, seg_pos={0 0}, upload=473975]: Done uploading piece. err: <nil>
Dec 28 03:56:37.362555102 (540h27m37.233638137s) [noise=true , address=77.90.31.11:4556 , piece_id=HAXLB4XPW3PVBIDQE4G736NCQ5INYDAYNDSM6R7YNXR5ZNSHAK5A, seg_pos={0 0}, upload=473975]: Done uploading piece. err: <nil>
Dec 28 03:56:37.430633264 (540h27m37.301716299s) [noise=true , address=45.91.116.5:4298 , piece_id=IVPUHKPLQQEHRLYOEAAOQ73HAYV5AGQH2EMRTZZUK3UESBPIXPQQ, seg_pos={0 0}, upload=473975]: Done uploading piece. err: <nil>
Dec 28 03:56:37.613195724 (540h27m37.484278759s) [noise=true , address=45.91.116.14:4191 , piece_id=BVVVHPL2V3AJHI3Q6JL5WTLHRNNRJFWDGCEOWQ66YS4GXIFZUDSQ, seg_pos={0 0}, upload=473975]: Done uploading piece. err: <nil>
Dec 28 03:56:37.656927147 (540h27m37.528010192s) [noise=true , address=45.91.116.7:4011 , piece_id=WDQTIVDOK6QHBE4XEUF2LYQR4ZVU6QBNPJTTIIC3RN2WUXCB34DA, seg_pos={0 0}, upload=473975]: Done uploading piece. err: <nil>
Dec 28 03:56:37.721331444 (540h27m37.592414479s) [noise=true , address=45.91.116.9:4264 , piece_id=J2SKQZEFPXF7ZUWQ2I55WR2NHE5AACWZASV2L3XCHUZ7KLEZ5WKQ, seg_pos={0 0}, upload=473975]: Done uploading piece. err: <nil>
Dec 28 03:56:38.022998270 (540h27m37.894081305s) [noise=true , address=77.90.31.14:4663 , piece_id=TIC7K6VF3UGW32ARXZ6ODCGZB3GTQIKI7KPPFQVSCXC3RKHNQABQ, seg_pos={0 0}, upload=473975]: Done uploading piece. err: <nil>
Dec 28 03:56:38.023015803 (540h27m37.894098838s) [seg_pos={0 0}, upload=473975]: Segment reached optimal threshold of 13 pieces.
Dec 28 03:56:38.023180326 (540h27m37.894263361s) [noise=true , address=145.79.227.41:4087 , piece_id=ZFNLKOERYVBTMBTQHUFFMEPTHUKLKQN3MMNGMPYF2BZH6YX7CU3Q, seg_pos={0 0}, upload=473975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR): protocol: write tcp 188.166.225.170:46956->145.79.227.41:4087: use of closed network connection; write tcp 188.166.225.170:46956->145.79.227.41:4087: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46956->145.79.227.41:4087: use of closed network connection
Dec 28 03:56:38.023186728 (540h27m37.894269773s) [noise=true , address=145.79.227.41:4087 , piece_id=ZFNLKOERYVBTMBTQHUFFMEPTHUKLKQN3MMNGMPYF2BZH6YX7CU3Q, seg_pos={0 0}, upload=473975]: Long tail cancellation due to optimal threshold
Dec 28 03:56:38.023307729 (540h27m37.894390804s) [noise=true , address=145.79.227.14:4840 , piece_id=GQB554ICYLZNOKKA3QOHSNY4D5S66GYHSM7U2BDI2GZ77RQWS5KQ, seg_pos={0 0}, upload=473975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: expected piece hash; context canceled; EOF
Dec 28 03:56:38.023316145 (540h27m37.89439918s) [noise=true , address=145.79.227.14:4840 , piece_id=GQB554ICYLZNOKKA3QOHSNY4D5S66GYHSM7U2BDI2GZ77RQWS5KQ, seg_pos={0 0}, upload=473975]: Long tail cancellation due to optimal threshold
Dec 28 03:56:38.023366070 (540h27m37.894449115s) [noise=true , address=77.90.31.15:4700 , piece_id=XJR5JONAZUEEALLJD3GTAO54OXXC5BEZRHARWZE2QD542TM7NZYQ, seg_pos={0 0}, upload=473975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: expected piece hash; context canceled; EOF
Dec 28 03:56:38.023382000 (540h27m37.894465036s) [noise=true , address=77.90.31.15:4700 , piece_id=XJR5JONAZUEEALLJD3GTAO54OXXC5BEZRHARWZE2QD542TM7NZYQ, seg_pos={0 0}, upload=473975]: Long tail cancellation due to optimal threshold
Dec 28 03:56:38.023420073 (540h27m37.894503118s) [seg_pos={0 0}, upload=473975]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 28 03:56:38.023510796 (540h27m37.894593831s) [upload=473975]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 03:56:38.023516126 (540h27m37.894599161s) [upload=473975]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 03:56:38.336523365 (540h27m38.2076064s) [upload=473975]: Done uploading segments...
Dec 28 03:56:38.336544726 (540h27m38.207627771s) [upload=473975]: Upload finished. err: <nil>
Duration: 30.003776193s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
20 out of 60 is failed (threshold=6) [....XX.....X..X.X....X.XXXX....X.X.X...X..X.X.X..X...XX.....] duration:
3.408511208s trace:
https://jaeger.dev.storj.io/trace/7e7d6d3ddfeaade0
FAIL (2025-12-28 03:57:02 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.408541377s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [...............X...X........................................] duration:
37.854306412s trace:
https://jaeger.dev.storj.io/trace/54422f75ff432c0
FAIL (2025-12-28 03:56:47 UTC)
orbiter warning:
operation: DeleteObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
, piece_id=SPQSUVHIHI32ZMIRE4NHSNJCMWS4NS6ZZPYF6FV7FQH6VADQIV5A, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:35.720215866 (539h54m35.591298911s) [noise=true , address=145.79.227.43:4187 , piece_id=MRILXYFSWVSH35ZCCFZWLEXENWJII6G6LOR2N277GIBMI4PXPPTQ, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:35.720567665 (539h54m35.5916507s) [noise=true , address=145.79.227.33:4896 , piece_id=7NRM7I4IDHW2WGOCG3B6L6HN74ANXWDJCFRPBM4BCHWJELSNNUXA, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:35.720883607 (539h54m35.591966652s) [noise=true , address=145.79.227.40:4062 , piece_id=RCVKQLDSWIWX2FQBTUFPEUXQZLP2C3ZCLV2JWHOV7CMJ7ZBNWZNQ, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:35.721874904 (539h54m35.592957949s) [noise=true , address=64.49.13.140:4513 , piece_id=Y3HW5I2FX2CWSPVUS7CU3JPNN2RHP2ASL757AUYINBKRKCLXAJGQ, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:35.722249658 (539h54m35.593332693s) [noise=true , address=64.49.13.169:4715 , piece_id=FRVNWFP2AE4DXE4RKZRE6WTK43T74BTTYM2U7E4P3B4XZIMC2QMQ, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:35.722786881 (539h54m35.593869926s) [noise=true , address=77.90.31.14:4673 , piece_id=ZIB5VOFL7PJSRFDV3HP5XAKYHOJXRVHCYDPIHKRYPNM5ROAY5DTQ, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:35.723112100 (539h54m35.594195135s) [noise=true , address=77.90.31.17:4779 , piece_id=TLU5ZLZRVAMGQLSE2LLNIUE4WOWJBMFJGFQKC3KOUJ23V3UVFMRA, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:35.723440756 (539h54m35.594523791s) [noise=true , address=145.79.227.37:4569 , piece_id=DUXSQUT36XUF66ZVQ2V7ZWNXA6JZKBDO5RXT3QVT26GE2FTH3QXQ, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:35.723849544 (539h54m35.594932589s) [noise=true , address=145.79.227.27:4723 , piece_id=4DECSGDNZLXNQVPU7QALLPFQYDO7MOG6TIBNWQ5UHNVCIHB4GUYQ, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:35.724241290 (539h54m35.595324335s) [noise=true , address=77.90.31.11:4587 , piece_id=6467VZCLWHJTZ36ENMEOEOU2E4MAUCDL2XHNAOMJTCCTUSN33NFA, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:35.724607808 (539h54m35.595690854s) [noise=true , address=145.79.227.47:4272 , piece_id=FL2U4W2YCV5ANDVFHB7H4TFT3SYTBGNLI3KZDI7MY2K6NTSMRUAA, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:35.724974035 (539h54m35.59605707s) [noise=true , address=145.79.227.18:4040 , piece_id=W55GK3QDHRHNR6YJ7JKVZMIEKFQLZSTXKSQ5OMW3U7XRI44EM6RA, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:35.725302281 (539h54m35.596385316s) [noise=true , address=145.79.227.38:4989 , piece_id=OUINZ6BXOLFCDNCXMX7PPFSKWMJ7IPZWO7DAGLGVV2AB7UHO3JYQ, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:35.726824420 (539h54m35.597907455s) [noise=true , address=145.79.227.26:4669 , piece_id=5D7SGVO274FEZ4JVESTMUYQ3RZ3DJA5DL2QZV5YKMXEB4TM6PHAA, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:37.063163473 (539h54m36.934246518s) [upload=473490]: Next returned nil segment
Dec 28 03:23:37.063176499 (539h54m36.934259534s) [upload=473490]: Waiting for error group managing segments...
Dec 28 03:23:37.416516311 (539h54m37.287599376s) [noise=true , address=64.49.13.140:4513 , piece_id=Y3HW5I2FX2CWSPVUS7CU3JPNN2RHP2ASL757AUYINBKRKCLXAJGQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil>
Dec 28 03:23:37.606629586 (539h54m37.477712622s) [noise=true , address=145.79.227.27:4723 , piece_id=4DECSGDNZLXNQVPU7QALLPFQYDO7MOG6TIBNWQ5UHNVCIHB4GUYQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil>
Dec 28 03:23:37.617730204 (539h54m37.488813249s) [noise=true , address=145.79.227.26:4669 , piece_id=5D7SGVO274FEZ4JVESTMUYQ3RZ3DJA5DL2QZV5YKMXEB4TM6PHAA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil>
Dec 28 03:23:37.632542950 (539h54m37.503625995s) [noise=true , address=145.79.227.38:4989 , piece_id=OUINZ6BXOLFCDNCXMX7PPFSKWMJ7IPZWO7DAGLGVV2AB7UHO3JYQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil>
Dec 28 03:23:37.643987531 (539h54m37.515070566s) [noise=true , address=145.79.227.47:4272 , piece_id=FL2U4W2YCV5ANDVFHB7H4TFT3SYTBGNLI3KZDI7MY2K6NTSMRUAA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil>
Dec 28 03:23:37.652913619 (539h54m37.523996654s) [noise=true , address=145.79.227.18:4040 , piece_id=W55GK3QDHRHNR6YJ7JKVZMIEKFQLZSTXKSQ5OMW3U7XRI44EM6RA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil>
Dec 28 03:23:37.661966357 (539h54m37.533049392s) [noise=true , address=145.79.227.37:4569 , piece_id=DUXSQUT36XUF66ZVQ2V7ZWNXA6JZKBDO5RXT3QVT26GE2FTH3QXQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil>
Dec 28 03:23:37.778627848 (539h54m37.649710893s) [noise=true , address=77.90.31.11:4561 , piece_id=HIUCZRRZOMU4RBYZU5SADJAOS4RVML2I6PILIX2V65RS22EBN42Q, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil>
Dec 28 03:23:37.815974812 (539h54m37.687057857s) [noise=true , address=45.91.116.14:4217 , piece_id=SPQSUVHIHI32ZMIRE4NHSNJCMWS4NS6ZZPYF6FV7FQH6VADQIV5A, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil>
Dec 28 03:23:37.861731780 (539h54m37.732814815s) [noise=true , address=77.90.31.14:4673 , piece_id=ZIB5VOFL7PJSRFDV3HP5XAKYHOJXRVHCYDPIHKRYPNM5ROAY5DTQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil>
Dec 28 03:23:38.093273211 (539h54m37.964356256s) [noise=true , address=77.90.31.11:4587 , piece_id=6467VZCLWHJTZ36ENMEOEOU2E4MAUCDL2XHNAOMJTCCTUSN33NFA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil>
Dec 28 03:23:38.267694395 (539h54m38.13877745s) [noise=true , address=77.90.31.17:4779 , piece_id=TLU5ZLZRVAMGQLSE2LLNIUE4WOWJBMFJGFQKC3KOUJ23V3UVFMRA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil>
Dec 28 03:23:38.540947761 (539h54m38.412030796s) [noise=true , address=145.79.227.40:4062 , piece_id=RCVKQLDSWIWX2FQBTUFPEUXQZLP2C3ZCLV2JWHOV7CMJ7ZBNWZNQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil>
Dec 28 03:23:38.815393527 (539h54m38.686476562s) [noise=true , address=145.79.227.43:4187 , piece_id=MRILXYFSWVSH35ZCCFZWLEXENWJII6G6LOR2N277GIBMI4PXPPTQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil>
Dec 28 03:23:38.815412894 (539h54m38.686495939s) [seg_pos={0 0}, upload=473490]: Segment reached optimal threshold of 14 pieces.
Dec 28 03:23:38.815594951 (539h54m38.686677986s) [noise=true , address=64.49.13.169:4715 , piece_id=FRVNWFP2AE4DXE4RKZRE6WTK43T74BTTYM2U7E4P3B4XZIMC2QMQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:43502->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:43502->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43502->64.49.13.169:4715: use of closed network connection
Dec 28 03:23:38.815601032 (539h54m38.686684077s) [noise=true , address=64.49.13.169:4715 , piece_id=FRVNWFP2AE4DXE4RKZRE6WTK43T74BTTYM2U7E4P3B4XZIMC2QMQ, seg_pos={0 0}, upload=473490]: Long tail cancellation due to optimal threshold
Dec 28 03:23:38.815826983 (539h54m38.686910018s) [noise=true , address=145.79.227.41:4118 , piece_id=C7JJ3NHR5355M64G2EF2WWERQVGWR243RQHDV3GBAIC3A66JDMHQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128w37UHxJB9R49iwmHbATt3bXUb1FMTQJ3Wjka7So7eqKWg4iF): protocol: expected piece hash; context canceled; EOF
Dec 28 03:23:38.815834507 (539h54m38.686917542s) [noise=true , address=145.79.227.41:4118 , piece_id=C7JJ3NHR5355M64G2EF2WWERQVGWR243RQHDV3GBAIC3A66JDMHQ, seg_pos={0 0}, upload=473490]: Long tail cancellation due to optimal threshold
Dec 28 03:23:38.815890433 (539h54m38.686973468s) [noise=true , address=145.79.227.33:4896 , piece_id=7NRM7I4IDHW2WGOCG3B6L6HN74ANXWDJCFRPBM4BCHWJELSNNUXA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ndSgjgfZMVQni5tw1ugdTrhJt4c4NagdvgYTbSbjatYYtfu1o): protocol: expected piece hash; context canceled; EOF
Dec 28 03:23:38.815914619 (539h54m38.686997654s) [noise=true , address=145.79.227.33:4896 , piece_id=7NRM7I4IDHW2WGOCG3B6L6HN74ANXWDJCFRPBM4BCHWJELSNNUXA, seg_pos={0 0}, upload=473490]: Long tail cancellation due to optimal threshold
Dec 28 03:23:38.816105613 (539h54m38.687188648s) [noise=true , address=145.79.227.28:4731 , piece_id=EPW3LUA3EWRCURHZ66NCRCA36CNIW4GYQMFNKPZZTRCB3YBGTKJA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NQzTwVYQa6ZzxAHQm1apkwWnqJstTnAJQyHKhBzC2jTN9kEVK): protocol: write tcp 188.166.225.170:58940->145.79.227.28:4731: use of closed network connection; write tcp 188.166.225.170:58940->145.79.227.28:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58940->145.79.227.28:4731: use of closed network connection
Dec 28 03:23:38.816112426 (539h54m38.687195461s) [noise=true , address=145.79.227.28:4731 , piece_id=EPW3LUA3EWRCURHZ66NCRCA36CNIW4GYQMFNKPZZTRCB3YBGTKJA, seg_pos={0 0}, upload=473490]: Long tail cancellation due to optimal threshold
Dec 28 03:23:38.816128436 (539h54m38.687211481s) [seg_pos={0 0}, upload=473490]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 28 03:23:38.816257582 (539h54m38.687340627s) [upload=473490]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 03:23:38.816265778 (539h54m38.687348813s) [upload=473490]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 03:23:39.111115407 (539h54m38.982198442s) [upload=473490]: Done uploading segments...
Dec 28 03:23:39.111138972 (539h54m38.982222008s) [upload=473490]: Upload finished. err: <nil>
Duration: 37.854323724s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [......................................X.....................] duration:
16.3641634s trace:
https://jaeger.dev.storj.io/trace/3b8130f32e60318d
FAIL (2025-12-28 03:56:46 UTC)
orbiter warning: context deadline exceeded
Duration: 16.364188399s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
53 out of 60 is failed (threshold=3) [XXXXXX.XXXXXXXXXXXXXXXXX.XXXXX.XXXXXXXXXXXXXXXXX..XXXX.XXXX.] duration:
5.852104199s trace:
https://jaeger.dev.storj.io/trace/150f497bcec75d
FAIL (2025-12-28 03:56:16 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: KNJV74VK3CWJK2LQEAB5SVPU5XZM537SMJGH72R4LO2MWC7D734Q) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: OWAG77TIAC5DPDW6XLNKWPG6RNLZD5JWDFOQYSBJS72JIYY6WCTA) context deadline exceeded
======LOGS======
ec 28 03:55:23.072561027 (540h26m22.943644072s) [upload=473956]: Starting upload
Dec 28 03:55:23.072622775 (540h26m22.94370582s) [upload=473956]: Done starting upload
Dec 28 03:55:23.072832945 (540h26m22.94391598s) [upload=473956]: Uploading segments...
Dec 28 03:55:23.072874283 (540h26m22.943957318s) [upload=473956]: Got next segment. Inline: false
Dec 28 03:55:23.072904250 (540h26m22.943987296s) [upload=473956]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 28 03:55:23.072908689 (540h26m22.943991734s) [upload=473956]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 28 03:55:23.387976121 (540h26m23.259059156s) [seg_pos={0 0}, upload=473956]: Begin upload segment...
Dec 28 03:55:23.388016939 (540h26m23.259099974s) [seg_pos={0 0}, upload=473956]: Done begin upload segment.
Dec 28 03:55:23.388040984 (540h26m23.259124029s) [noise=true , address=145.79.227.36:4519 , piece_id=NMYQ3ZFAAV6KZD64YO4QFEVWTKFXF5M5FSRCRZHFFFACPVMUUEEQ, seg_pos={0 0}, upload=473956]: Uploading piece...
Dec 28 03:55:23.388073446 (540h26m23.259156491s) [noise=true , address=77.90.31.15:4700 , piece_id=OWAG77TIAC5DPDW6XLNKWPG6RNLZD5JWDFOQYSBJS72JIYY6WCTA, seg_pos={0 0}, upload=473956]: Uploading piece...
Dec 28 03:55:23.388627441 (540h26m23.259710476s) [noise=true , address=145.79.227.52:4445 , piece_id=BPCT6SAYOOO5UA4RLRFDESPRY4SQEAMVJCCY357CP5MX2LJKOV2A, seg_pos={0 0}, upload=473956]: Uploading piece...
Dec 28 03:55:23.388714467 (540h26m23.259797513s) [noise=true , address=45.91.116.7:4010 , piece_id=MRYTVWTBBE2SZMYM3SBAZB4YBMSDIRYQ5QQWRHCZHQFNUAY6EXEQ, seg_pos={0 0}, upload=473956]: Uploading piece...
Dec 28 03:55:23.389117384 (540h26m23.260200429s) [noise=true , address=145.79.227.17:4970 , piece_id=D2FJMF23XF7MJICJGQ5MT225XGEGEXPO2I56CC7PYPBBVVSLNSJA, seg_pos={0 0}, upload=473956]: Uploading piece...
Dec 28 03:55:23.389495124 (540h26m23.260578169s) [noise=true , address=145.79.227.45:4230 , piece_id=P3YOHIYAXFYL2QEEVSWBHJCNUQNURDIHLJW5RDA3CCE2BTAZNEDQ, seg_pos={0 0}, upload=473956]: Uploading piece...
Dec 28 03:55:23.389664296 (540h26m23.260747341s) [noise=true , address=145.79.227.46:4602 , piece_id=F33JSDA4GRIK2YYS6SO3TDHAAK55S3UVOZFRIZVIR4KBIYDU2QVA, seg_pos={0 0}, upload=473956]: Uploading piece...
Dec 28 03:55:23.389942265 (540h26m23.2610253s) [noise=true , address=45.91.116.14:4190 , piece_id=KNJV74VK3CWJK2LQEAB5SVPU5XZM537SMJGH72R4LO2MWC7D734Q, seg_pos={0 0}, upload=473956]: Uploading piece...
Dec 28 03:55:23.390124021 (540h26m23.261207056s) [noise=true , address=145.79.227.19:4078 , piece_id=FIPOP4XNJ7D4VLOK3NXUV4KGQO563LDOGGYS57NLTRU4GR5WYQ7A, seg_pos={0 0}, upload=473956]: Uploading piece...
Dec 28 03:55:23.390388495 (540h26m23.26147153s) [noise=true , address=45.91.116.5:4300 , piece_id=VL7V3MWFHJXFHFCMGRGW5PTB7FLAEJRK5GYTQ2X5VK7SOPWHQ5LQ, seg_pos={0 0}, upload=473956]: Uploading piece...
Dec 28 03:55:23.390696001 (540h26m23.261779036s) [noise=true , address=145.79.227.47:4267 , piece_id=HVA5LP4LLKPCTYZD7GXSLGARXBEP6LSBGWWMLE75TBWLMP2CXKIA, seg_pos={0 0}, upload=473956]: Uploading piece...
Dec 28 03:55:23.391070383 (540h26m23.262153418s) [noise=true , address=77.90.31.14:4661 , piece_id=NZGSLU5J527AB4Y4CBQR5XTZ7KFDKRR2XZWFMWPUFAA77GI2IJAQ, seg_pos={0 0}, upload=473956]: Uploading piece...
Dec 28 03:55:23.391196463 (540h26m23.262279498s) [noise=true , address=145.79.227.48:4301 , piece_id=R66XL5K3R6KE2GE7AVMKIRHBF4Z5Z4V7W3IA3FETZDOLNCQ4P2JA, seg_pos={0 0}, upload=473956]: Uploading piece...
Dec 28 03:55:23.391498960 (540h26m23.262582005s) [noise=true , address=145.79.227.16:4909 , piece_id=VBTA33MPR64447LDH6QJBRBRNB2EFIEW5T4O2N73M67F63B66MIQ, seg_pos={0 0}, upload=473956]: Uploading piece...
Dec 28 03:55:23.391730810 (540h26m23.262813845s) [noise=true , address=145.79.227.41:4085 , piece_id=I4SPBKOHRDVWNQ7UH6IGVGJ2UVOMNMCGSTLDZ6AO6GKRT26SA6HQ, seg_pos={0 0}, upload=473956]: Uploading piece...
Dec 28 03:55:23.392314702 (540h26m23.263397748s) [noise=true , address=145.79.227.31:4797 , piece_id=7ELEUCZG6JGAFFVQYPOHZ7BTZIHU4NIIGNOJLYZ6COLK3YGHRLFA, seg_pos={0 0}, upload=473956]: Uploading piece...
Dec 28 03:55:25.079237864 (540h26m24.950320899s) [upload=473956]: Next returned nil segment
Dec 28 03:55:25.079252191 (540h26m24.950335237s) [upload=473956]: Waiting for error group managing segments...
Dec 28 03:55:25.392768049 (540h26m25.263851095s) [noise=true , address=145.79.227.48:4301 , piece_id=R66XL5K3R6KE2GE7AVMKIRHBF4Z5Z4V7W3IA3FETZDOLNCQ4P2JA, seg_pos={0 0}, upload=473956]: Done uploading piece. err: <nil>
Dec 28 03:55:25.414606793 (540h26m25.285689828s) [noise=true , address=145.79.227.31:4797 , piece_id=7ELEUCZG6JGAFFVQYPOHZ7BTZIHU4NIIGNOJLYZ6COLK3YGHRLFA, seg_pos={0 0}, upload=473956]: Done uploading piece. err: <nil>
Dec 28 03:55:25.416924276 (540h26m25.288007321s) [noise=true , address=145.79.227.41:4085 , piece_id=I4SPBKOHRDVWNQ7UH6IGVGJ2UVOMNMCGSTLDZ6AO6GKRT26SA6HQ, seg_pos={0 0}, upload=473956]: Done uploading piece. err: <nil>
Dec 28 03:55:25.417989104 (540h26m25.289072139s) [noise=true , address=145.79.227.46:4602 , piece_id=F33JSDA4GRIK2YYS6SO3TDHAAK55S3UVOZFRIZVIR4KBIYDU2QVA, seg_pos={0 0}, upload=473956]: Done uploading piece. err: <nil>
Dec 28 03:55:25.423153919 (540h26m25.294236964s) [noise=true , address=145.79.227.17:4970 , piece_id=D2FJMF23XF7MJICJGQ5MT225XGEGEXPO2I56CC7PYPBBVVSLNSJA, seg_pos={0 0}, upload=473956]: Done uploading piece. err: <nil>
Dec 28 03:55:25.636367464 (540h26m25.507450519s) [noise=true , address=45.91.116.7:4010 , piece_id=MRYTVWTBBE2SZMYM3SBAZB4YBMSDIRYQ5QQWRHCZHQFNUAY6EXEQ, seg_pos={0 0}, upload=473956]: Done uploading piece. err: <nil>
Dec 28 03:55:25.661527754 (540h26m25.532610799s) [noise=true , address=77.90.31.15:4700 , piece_id=OWAG77TIAC5DPDW6XLNKWPG6RNLZD5JWDFOQYSBJS72JIYY6WCTA, seg_pos={0 0}, upload=473956]: Done uploading piece. err: <nil>
Dec 28 03:55:25.665493916 (540h26m25.536576961s) [noise=true , address=45.91.116.14:4190 , piece_id=KNJV74VK3CWJK2LQEAB5SVPU5XZM537SMJGH72R4LO2MWC7D734Q, seg_pos={0 0}, upload=473956]: Done uploading piece. err: <nil>
Dec 28 03:55:25.805291183 (540h26m25.676374218s) [noise=true , address=45.91.116.5:4300 , piece_id=VL7V3MWFHJXFHFCMGRGW5PTB7FLAEJRK5GYTQ2X5VK7SOPWHQ5LQ, seg_pos={0 0}, upload=473956]: Done uploading piece. err: <nil>
Dec 28 03:55:25.877831202 (540h26m25.748914247s) [noise=true , address=145.79.227.36:4519 , piece_id=NMYQ3ZFAAV6KZD64YO4QFEVWTKFXF5M5FSRCRZHFFFACPVMUUEEQ, seg_pos={0 0}, upload=473956]: Done uploading piece. err: <nil>
Dec 28 03:55:25.888987575 (540h26m25.76007061s) [noise=true , address=145.79.227.19:4078 , piece_id=FIPOP4XNJ7D4VLOK3NXUV4KGQO563LDOGGYS57NLTRU4GR5WYQ7A, seg_pos={0 0}, upload=473956]: Done uploading piece. err: <nil>
Dec 28 03:55:25.890395467 (540h26m25.761478502s) [noise=true , address=145.79.227.47:4267 , piece_id=HVA5LP4LLKPCTYZD7GXSLGARXBEP6LSBGWWMLE75TBWLMP2CXKIA, seg_pos={0 0}, upload=473956]: Done uploading piece. err: <nil>
Dec 28 03:55:25.890982595 (540h26m25.76206564s) [noise=true , address=145.79.227.45:4230 , piece_id=P3YOHIYAXFYL2QEEVSWBHJCNUQNURDIHLJW5RDA3CCE2BTAZNEDQ, seg_pos={0 0}, upload=473956]: Done uploading piece. err: <nil>
Dec 28 03:55:25.891002452 (540h26m25.762085497s) [seg_pos={0 0}, upload=473956]: Segment reached optimal threshold of 13 pieces.
Dec 28 03:55:25.891225377 (540h26m25.762308422s) [noise=true , address=145.79.227.52:4445 , piece_id=BPCT6SAYOOO5UA4RLRFDESPRY4SQEAMVJCCY357CP5MX2LJKOV2A, seg_pos={0 0}, upload=473956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12vXrkhLqCofhR4Fmfqb98gxnqWQF6vMV4q3ATbRvPfcqi82WSo): protocol: expected piece hash; context canceled; EOF
Dec 28 03:55:25.891235486 (540h26m25.762318521s) [noise=true , address=145.79.227.52:4445 , piece_id=BPCT6SAYOOO5UA4RLRFDESPRY4SQEAMVJCCY357CP5MX2LJKOV2A, seg_pos={0 0}, upload=473956]: Long tail cancellation due to optimal threshold
Dec 28 03:55:25.891365414 (540h26m25.762448449s) [noise=true , address=77.90.31.14:4661 , piece_id=NZGSLU5J527AB4Y4CBQR5XTZ7KFDKRR2XZWFMWPUFAA77GI2IJAQ, seg_pos={0 0}, upload=473956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: write tcp 188.166.225.170:38440->77.90.31.14:4661: use of closed network connection; write tcp 188.166.225.170:38440->77.90.31.14:4661: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38440->77.90.31.14:4661: use of closed network connection
Dec 28 03:55:25.891371205 (540h26m25.76245425s) [noise=true , address=77.90.31.14:4661 , piece_id=NZGSLU5J527AB4Y4CBQR5XTZ7KFDKRR2XZWFMWPUFAA77GI2IJAQ, seg_pos={0 0}, upload=473956]: Long tail cancellation due to optimal threshold
Dec 28 03:55:25.891655446 (540h26m25.762738501s) [noise=true , address=145.79.227.16:4909 , piece_id=VBTA33MPR64447LDH6QJBRBRNB2EFIEW5T4O2N73M67F63B66MIQ, seg_pos={0 0}, upload=473956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x): protocol: expected piece hash; context canceled; EOF
Dec 28 03:55:25.891668570 (540h26m25.762751615s) [noise=true , address=145.79.227.16:4909 , piece_id=VBTA33MPR64447LDH6QJBRBRNB2EFIEW5T4O2N73M67F63B66MIQ, seg_pos={0 0}, upload=473956]: Long tail cancellation due to optimal threshold
Dec 28 03:55:25.891693769 (540h26m25.762776814s) [seg_pos={0 0}, upload=473956]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 28 03:55:25.891802816 (540h26m25.762885861s) [upload=473956]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 03:55:25.891810081 (540h26m25.762893126s) [upload=473956]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 03:55:26.194494122 (540h26m26.065577157s) [upload=473956]: Done uploading segments...
Dec 28 03:55:26.194520563 (540h26m26.065603598s) [upload=473956]: Upload finished. err: <nil>
Duration: 5.852140167s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [................X...X.......................................] duration:
42.160440895s trace:
https://jaeger.dev.storj.io/trace/54422f75ff432c0
FAIL (2025-12-28 03:56:06 UTC)
orbiter warning:
operation: DeleteObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
, piece_id=SPQSUVHIHI32ZMIRE4NHSNJCMWS4NS6ZZPYF6FV7FQH6VADQIV5A, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:35.720215866 (539h54m35.591298911s) [noise=true , address=145.79.227.43:4187 , piece_id=MRILXYFSWVSH35ZCCFZWLEXENWJII6G6LOR2N277GIBMI4PXPPTQ, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:35.720567665 (539h54m35.5916507s) [noise=true , address=145.79.227.33:4896 , piece_id=7NRM7I4IDHW2WGOCG3B6L6HN74ANXWDJCFRPBM4BCHWJELSNNUXA, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:35.720883607 (539h54m35.591966652s) [noise=true , address=145.79.227.40:4062 , piece_id=RCVKQLDSWIWX2FQBTUFPEUXQZLP2C3ZCLV2JWHOV7CMJ7ZBNWZNQ, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:35.721874904 (539h54m35.592957949s) [noise=true , address=64.49.13.140:4513 , piece_id=Y3HW5I2FX2CWSPVUS7CU3JPNN2RHP2ASL757AUYINBKRKCLXAJGQ, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:35.722249658 (539h54m35.593332693s) [noise=true , address=64.49.13.169:4715 , piece_id=FRVNWFP2AE4DXE4RKZRE6WTK43T74BTTYM2U7E4P3B4XZIMC2QMQ, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:35.722786881 (539h54m35.593869926s) [noise=true , address=77.90.31.14:4673 , piece_id=ZIB5VOFL7PJSRFDV3HP5XAKYHOJXRVHCYDPIHKRYPNM5ROAY5DTQ, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:35.723112100 (539h54m35.594195135s) [noise=true , address=77.90.31.17:4779 , piece_id=TLU5ZLZRVAMGQLSE2LLNIUE4WOWJBMFJGFQKC3KOUJ23V3UVFMRA, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:35.723440756 (539h54m35.594523791s) [noise=true , address=145.79.227.37:4569 , piece_id=DUXSQUT36XUF66ZVQ2V7ZWNXA6JZKBDO5RXT3QVT26GE2FTH3QXQ, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:35.723849544 (539h54m35.594932589s) [noise=true , address=145.79.227.27:4723 , piece_id=4DECSGDNZLXNQVPU7QALLPFQYDO7MOG6TIBNWQ5UHNVCIHB4GUYQ, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:35.724241290 (539h54m35.595324335s) [noise=true , address=77.90.31.11:4587 , piece_id=6467VZCLWHJTZ36ENMEOEOU2E4MAUCDL2XHNAOMJTCCTUSN33NFA, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:35.724607808 (539h54m35.595690854s) [noise=true , address=145.79.227.47:4272 , piece_id=FL2U4W2YCV5ANDVFHB7H4TFT3SYTBGNLI3KZDI7MY2K6NTSMRUAA, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:35.724974035 (539h54m35.59605707s) [noise=true , address=145.79.227.18:4040 , piece_id=W55GK3QDHRHNR6YJ7JKVZMIEKFQLZSTXKSQ5OMW3U7XRI44EM6RA, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:35.725302281 (539h54m35.596385316s) [noise=true , address=145.79.227.38:4989 , piece_id=OUINZ6BXOLFCDNCXMX7PPFSKWMJ7IPZWO7DAGLGVV2AB7UHO3JYQ, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:35.726824420 (539h54m35.597907455s) [noise=true , address=145.79.227.26:4669 , piece_id=5D7SGVO274FEZ4JVESTMUYQ3RZ3DJA5DL2QZV5YKMXEB4TM6PHAA, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:37.063163473 (539h54m36.934246518s) [upload=473490]: Next returned nil segment
Dec 28 03:23:37.063176499 (539h54m36.934259534s) [upload=473490]: Waiting for error group managing segments...
Dec 28 03:23:37.416516311 (539h54m37.287599376s) [noise=true , address=64.49.13.140:4513 , piece_id=Y3HW5I2FX2CWSPVUS7CU3JPNN2RHP2ASL757AUYINBKRKCLXAJGQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil>
Dec 28 03:23:37.606629586 (539h54m37.477712622s) [noise=true , address=145.79.227.27:4723 , piece_id=4DECSGDNZLXNQVPU7QALLPFQYDO7MOG6TIBNWQ5UHNVCIHB4GUYQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil>
Dec 28 03:23:37.617730204 (539h54m37.488813249s) [noise=true , address=145.79.227.26:4669 , piece_id=5D7SGVO274FEZ4JVESTMUYQ3RZ3DJA5DL2QZV5YKMXEB4TM6PHAA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil>
Dec 28 03:23:37.632542950 (539h54m37.503625995s) [noise=true , address=145.79.227.38:4989 , piece_id=OUINZ6BXOLFCDNCXMX7PPFSKWMJ7IPZWO7DAGLGVV2AB7UHO3JYQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil>
Dec 28 03:23:37.643987531 (539h54m37.515070566s) [noise=true , address=145.79.227.47:4272 , piece_id=FL2U4W2YCV5ANDVFHB7H4TFT3SYTBGNLI3KZDI7MY2K6NTSMRUAA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil>
Dec 28 03:23:37.652913619 (539h54m37.523996654s) [noise=true , address=145.79.227.18:4040 , piece_id=W55GK3QDHRHNR6YJ7JKVZMIEKFQLZSTXKSQ5OMW3U7XRI44EM6RA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil>
Dec 28 03:23:37.661966357 (539h54m37.533049392s) [noise=true , address=145.79.227.37:4569 , piece_id=DUXSQUT36XUF66ZVQ2V7ZWNXA6JZKBDO5RXT3QVT26GE2FTH3QXQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil>
Dec 28 03:23:37.778627848 (539h54m37.649710893s) [noise=true , address=77.90.31.11:4561 , piece_id=HIUCZRRZOMU4RBYZU5SADJAOS4RVML2I6PILIX2V65RS22EBN42Q, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil>
Dec 28 03:23:37.815974812 (539h54m37.687057857s) [noise=true , address=45.91.116.14:4217 , piece_id=SPQSUVHIHI32ZMIRE4NHSNJCMWS4NS6ZZPYF6FV7FQH6VADQIV5A, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil>
Dec 28 03:23:37.861731780 (539h54m37.732814815s) [noise=true , address=77.90.31.14:4673 , piece_id=ZIB5VOFL7PJSRFDV3HP5XAKYHOJXRVHCYDPIHKRYPNM5ROAY5DTQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil>
Dec 28 03:23:38.093273211 (539h54m37.964356256s) [noise=true , address=77.90.31.11:4587 , piece_id=6467VZCLWHJTZ36ENMEOEOU2E4MAUCDL2XHNAOMJTCCTUSN33NFA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil>
Dec 28 03:23:38.267694395 (539h54m38.13877745s) [noise=true , address=77.90.31.17:4779 , piece_id=TLU5ZLZRVAMGQLSE2LLNIUE4WOWJBMFJGFQKC3KOUJ23V3UVFMRA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil>
Dec 28 03:23:38.540947761 (539h54m38.412030796s) [noise=true , address=145.79.227.40:4062 , piece_id=RCVKQLDSWIWX2FQBTUFPEUXQZLP2C3ZCLV2JWHOV7CMJ7ZBNWZNQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil>
Dec 28 03:23:38.815393527 (539h54m38.686476562s) [noise=true , address=145.79.227.43:4187 , piece_id=MRILXYFSWVSH35ZCCFZWLEXENWJII6G6LOR2N277GIBMI4PXPPTQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil>
Dec 28 03:23:38.815412894 (539h54m38.686495939s) [seg_pos={0 0}, upload=473490]: Segment reached optimal threshold of 14 pieces.
Dec 28 03:23:38.815594951 (539h54m38.686677986s) [noise=true , address=64.49.13.169:4715 , piece_id=FRVNWFP2AE4DXE4RKZRE6WTK43T74BTTYM2U7E4P3B4XZIMC2QMQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:43502->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:43502->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43502->64.49.13.169:4715: use of closed network connection
Dec 28 03:23:38.815601032 (539h54m38.686684077s) [noise=true , address=64.49.13.169:4715 , piece_id=FRVNWFP2AE4DXE4RKZRE6WTK43T74BTTYM2U7E4P3B4XZIMC2QMQ, seg_pos={0 0}, upload=473490]: Long tail cancellation due to optimal threshold
Dec 28 03:23:38.815826983 (539h54m38.686910018s) [noise=true , address=145.79.227.41:4118 , piece_id=C7JJ3NHR5355M64G2EF2WWERQVGWR243RQHDV3GBAIC3A66JDMHQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128w37UHxJB9R49iwmHbATt3bXUb1FMTQJ3Wjka7So7eqKWg4iF): protocol: expected piece hash; context canceled; EOF
Dec 28 03:23:38.815834507 (539h54m38.686917542s) [noise=true , address=145.79.227.41:4118 , piece_id=C7JJ3NHR5355M64G2EF2WWERQVGWR243RQHDV3GBAIC3A66JDMHQ, seg_pos={0 0}, upload=473490]: Long tail cancellation due to optimal threshold
Dec 28 03:23:38.815890433 (539h54m38.686973468s) [noise=true , address=145.79.227.33:4896 , piece_id=7NRM7I4IDHW2WGOCG3B6L6HN74ANXWDJCFRPBM4BCHWJELSNNUXA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ndSgjgfZMVQni5tw1ugdTrhJt4c4NagdvgYTbSbjatYYtfu1o): protocol: expected piece hash; context canceled; EOF
Dec 28 03:23:38.815914619 (539h54m38.686997654s) [noise=true , address=145.79.227.33:4896 , piece_id=7NRM7I4IDHW2WGOCG3B6L6HN74ANXWDJCFRPBM4BCHWJELSNNUXA, seg_pos={0 0}, upload=473490]: Long tail cancellation due to optimal threshold
Dec 28 03:23:38.816105613 (539h54m38.687188648s) [noise=true , address=145.79.227.28:4731 , piece_id=EPW3LUA3EWRCURHZ66NCRCA36CNIW4GYQMFNKPZZTRCB3YBGTKJA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NQzTwVYQa6ZzxAHQm1apkwWnqJstTnAJQyHKhBzC2jTN9kEVK): protocol: write tcp 188.166.225.170:58940->145.79.227.28:4731: use of closed network connection; write tcp 188.166.225.170:58940->145.79.227.28:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58940->145.79.227.28:4731: use of closed network connection
Dec 28 03:23:38.816112426 (539h54m38.687195461s) [noise=true , address=145.79.227.28:4731 , piece_id=EPW3LUA3EWRCURHZ66NCRCA36CNIW4GYQMFNKPZZTRCB3YBGTKJA, seg_pos={0 0}, upload=473490]: Long tail cancellation due to optimal threshold
Dec 28 03:23:38.816128436 (539h54m38.687211481s) [seg_pos={0 0}, upload=473490]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 28 03:23:38.816257582 (539h54m38.687340627s) [upload=473490]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 03:23:38.816265778 (539h54m38.687348813s) [upload=473490]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 03:23:39.111115407 (539h54m38.982198442s) [upload=473490]: Done uploading segments...
Dec 28 03:23:39.111138972 (539h54m38.982222008s) [upload=473490]: Upload finished. err: <nil>
Duration: 42.160482986s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.......................................X....................] duration:
15.762942052s trace:
https://jaeger.dev.storj.io/trace/3b8130f32e60318d
FAIL (2025-12-28 03:56:02 UTC)
orbiter warning: context deadline exceeded
Duration: 15.762964455s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
20 out of 60 is failed (threshold=6) [.....XX.....X..X.X....X.XXXX....X.X.X...X..X.X.X..X...XX....] duration:
5.000692557s trace:
https://jaeger.dev.storj.io/trace/7e7d6d3ddfeaade0
FAIL (2025-12-28 03:55:59 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.000716963s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
53 out of 60 is failed (threshold=3) [.XXXXXX.XXXXXXXXXXXXXXXXX.XXXXX.XXXXXXXXXXXXXXXXX..XXXX.XXXX] duration:
30.00316076s trace:
https://jaeger.dev.storj.io/trace/150f497bcec75d
FAIL (2025-12-28 03:55:52 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: KNJV74VK3CWJK2LQEAB5SVPU5XZM537SMJGH72R4LO2MWC7D734Q) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: OWAG77TIAC5DPDW6XLNKWPG6RNLZD5JWDFOQYSBJS72JIYY6WCTA) context deadline exceeded
======LOGS======
ec 28 03:55:23.072561027 (540h26m22.943644072s) [upload=473956]: Starting upload
Dec 28 03:55:23.072622775 (540h26m22.94370582s) [upload=473956]: Done starting upload
Dec 28 03:55:23.072832945 (540h26m22.94391598s) [upload=473956]: Uploading segments...
Dec 28 03:55:23.072874283 (540h26m22.943957318s) [upload=473956]: Got next segment. Inline: false
Dec 28 03:55:23.072904250 (540h26m22.943987296s) [upload=473956]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 28 03:55:23.072908689 (540h26m22.943991734s) [upload=473956]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 28 03:55:23.387976121 (540h26m23.259059156s) [seg_pos={0 0}, upload=473956]: Begin upload segment...
Dec 28 03:55:23.388016939 (540h26m23.259099974s) [seg_pos={0 0}, upload=473956]: Done begin upload segment.
Dec 28 03:55:23.388040984 (540h26m23.259124029s) [noise=true , address=145.79.227.36:4519 , piece_id=NMYQ3ZFAAV6KZD64YO4QFEVWTKFXF5M5FSRCRZHFFFACPVMUUEEQ, seg_pos={0 0}, upload=473956]: Uploading piece...
Dec 28 03:55:23.388073446 (540h26m23.259156491s) [noise=true , address=77.90.31.15:4700 , piece_id=OWAG77TIAC5DPDW6XLNKWPG6RNLZD5JWDFOQYSBJS72JIYY6WCTA, seg_pos={0 0}, upload=473956]: Uploading piece...
Dec 28 03:55:23.388627441 (540h26m23.259710476s) [noise=true , address=145.79.227.52:4445 , piece_id=BPCT6SAYOOO5UA4RLRFDESPRY4SQEAMVJCCY357CP5MX2LJKOV2A, seg_pos={0 0}, upload=473956]: Uploading piece...
Dec 28 03:55:23.388714467 (540h26m23.259797513s) [noise=true , address=45.91.116.7:4010 , piece_id=MRYTVWTBBE2SZMYM3SBAZB4YBMSDIRYQ5QQWRHCZHQFNUAY6EXEQ, seg_pos={0 0}, upload=473956]: Uploading piece...
Dec 28 03:55:23.389117384 (540h26m23.260200429s) [noise=true , address=145.79.227.17:4970 , piece_id=D2FJMF23XF7MJICJGQ5MT225XGEGEXPO2I56CC7PYPBBVVSLNSJA, seg_pos={0 0}, upload=473956]: Uploading piece...
Dec 28 03:55:23.389495124 (540h26m23.260578169s) [noise=true , address=145.79.227.45:4230 , piece_id=P3YOHIYAXFYL2QEEVSWBHJCNUQNURDIHLJW5RDA3CCE2BTAZNEDQ, seg_pos={0 0}, upload=473956]: Uploading piece...
Dec 28 03:55:23.389664296 (540h26m23.260747341s) [noise=true , address=145.79.227.46:4602 , piece_id=F33JSDA4GRIK2YYS6SO3TDHAAK55S3UVOZFRIZVIR4KBIYDU2QVA, seg_pos={0 0}, upload=473956]: Uploading piece...
Dec 28 03:55:23.389942265 (540h26m23.2610253s) [noise=true , address=45.91.116.14:4190 , piece_id=KNJV74VK3CWJK2LQEAB5SVPU5XZM537SMJGH72R4LO2MWC7D734Q, seg_pos={0 0}, upload=473956]: Uploading piece...
Dec 28 03:55:23.390124021 (540h26m23.261207056s) [noise=true , address=145.79.227.19:4078 , piece_id=FIPOP4XNJ7D4VLOK3NXUV4KGQO563LDOGGYS57NLTRU4GR5WYQ7A, seg_pos={0 0}, upload=473956]: Uploading piece...
Dec 28 03:55:23.390388495 (540h26m23.26147153s) [noise=true , address=45.91.116.5:4300 , piece_id=VL7V3MWFHJXFHFCMGRGW5PTB7FLAEJRK5GYTQ2X5VK7SOPWHQ5LQ, seg_pos={0 0}, upload=473956]: Uploading piece...
Dec 28 03:55:23.390696001 (540h26m23.261779036s) [noise=true , address=145.79.227.47:4267 , piece_id=HVA5LP4LLKPCTYZD7GXSLGARXBEP6LSBGWWMLE75TBWLMP2CXKIA, seg_pos={0 0}, upload=473956]: Uploading piece...
Dec 28 03:55:23.391070383 (540h26m23.262153418s) [noise=true , address=77.90.31.14:4661 , piece_id=NZGSLU5J527AB4Y4CBQR5XTZ7KFDKRR2XZWFMWPUFAA77GI2IJAQ, seg_pos={0 0}, upload=473956]: Uploading piece...
Dec 28 03:55:23.391196463 (540h26m23.262279498s) [noise=true , address=145.79.227.48:4301 , piece_id=R66XL5K3R6KE2GE7AVMKIRHBF4Z5Z4V7W3IA3FETZDOLNCQ4P2JA, seg_pos={0 0}, upload=473956]: Uploading piece...
Dec 28 03:55:23.391498960 (540h26m23.262582005s) [noise=true , address=145.79.227.16:4909 , piece_id=VBTA33MPR64447LDH6QJBRBRNB2EFIEW5T4O2N73M67F63B66MIQ, seg_pos={0 0}, upload=473956]: Uploading piece...
Dec 28 03:55:23.391730810 (540h26m23.262813845s) [noise=true , address=145.79.227.41:4085 , piece_id=I4SPBKOHRDVWNQ7UH6IGVGJ2UVOMNMCGSTLDZ6AO6GKRT26SA6HQ, seg_pos={0 0}, upload=473956]: Uploading piece...
Dec 28 03:55:23.392314702 (540h26m23.263397748s) [noise=true , address=145.79.227.31:4797 , piece_id=7ELEUCZG6JGAFFVQYPOHZ7BTZIHU4NIIGNOJLYZ6COLK3YGHRLFA, seg_pos={0 0}, upload=473956]: Uploading piece...
Dec 28 03:55:25.079237864 (540h26m24.950320899s) [upload=473956]: Next returned nil segment
Dec 28 03:55:25.079252191 (540h26m24.950335237s) [upload=473956]: Waiting for error group managing segments...
Dec 28 03:55:25.392768049 (540h26m25.263851095s) [noise=true , address=145.79.227.48:4301 , piece_id=R66XL5K3R6KE2GE7AVMKIRHBF4Z5Z4V7W3IA3FETZDOLNCQ4P2JA, seg_pos={0 0}, upload=473956]: Done uploading piece. err: <nil>
Dec 28 03:55:25.414606793 (540h26m25.285689828s) [noise=true , address=145.79.227.31:4797 , piece_id=7ELEUCZG6JGAFFVQYPOHZ7BTZIHU4NIIGNOJLYZ6COLK3YGHRLFA, seg_pos={0 0}, upload=473956]: Done uploading piece. err: <nil>
Dec 28 03:55:25.416924276 (540h26m25.288007321s) [noise=true , address=145.79.227.41:4085 , piece_id=I4SPBKOHRDVWNQ7UH6IGVGJ2UVOMNMCGSTLDZ6AO6GKRT26SA6HQ, seg_pos={0 0}, upload=473956]: Done uploading piece. err: <nil>
Dec 28 03:55:25.417989104 (540h26m25.289072139s) [noise=true , address=145.79.227.46:4602 , piece_id=F33JSDA4GRIK2YYS6SO3TDHAAK55S3UVOZFRIZVIR4KBIYDU2QVA, seg_pos={0 0}, upload=473956]: Done uploading piece. err: <nil>
Dec 28 03:55:25.423153919 (540h26m25.294236964s) [noise=true , address=145.79.227.17:4970 , piece_id=D2FJMF23XF7MJICJGQ5MT225XGEGEXPO2I56CC7PYPBBVVSLNSJA, seg_pos={0 0}, upload=473956]: Done uploading piece. err: <nil>
Dec 28 03:55:25.636367464 (540h26m25.507450519s) [noise=true , address=45.91.116.7:4010 , piece_id=MRYTVWTBBE2SZMYM3SBAZB4YBMSDIRYQ5QQWRHCZHQFNUAY6EXEQ, seg_pos={0 0}, upload=473956]: Done uploading piece. err: <nil>
Dec 28 03:55:25.661527754 (540h26m25.532610799s) [noise=true , address=77.90.31.15:4700 , piece_id=OWAG77TIAC5DPDW6XLNKWPG6RNLZD5JWDFOQYSBJS72JIYY6WCTA, seg_pos={0 0}, upload=473956]: Done uploading piece. err: <nil>
Dec 28 03:55:25.665493916 (540h26m25.536576961s) [noise=true , address=45.91.116.14:4190 , piece_id=KNJV74VK3CWJK2LQEAB5SVPU5XZM537SMJGH72R4LO2MWC7D734Q, seg_pos={0 0}, upload=473956]: Done uploading piece. err: <nil>
Dec 28 03:55:25.805291183 (540h26m25.676374218s) [noise=true , address=45.91.116.5:4300 , piece_id=VL7V3MWFHJXFHFCMGRGW5PTB7FLAEJRK5GYTQ2X5VK7SOPWHQ5LQ, seg_pos={0 0}, upload=473956]: Done uploading piece. err: <nil>
Dec 28 03:55:25.877831202 (540h26m25.748914247s) [noise=true , address=145.79.227.36:4519 , piece_id=NMYQ3ZFAAV6KZD64YO4QFEVWTKFXF5M5FSRCRZHFFFACPVMUUEEQ, seg_pos={0 0}, upload=473956]: Done uploading piece. err: <nil>
Dec 28 03:55:25.888987575 (540h26m25.76007061s) [noise=true , address=145.79.227.19:4078 , piece_id=FIPOP4XNJ7D4VLOK3NXUV4KGQO563LDOGGYS57NLTRU4GR5WYQ7A, seg_pos={0 0}, upload=473956]: Done uploading piece. err: <nil>
Dec 28 03:55:25.890395467 (540h26m25.761478502s) [noise=true , address=145.79.227.47:4267 , piece_id=HVA5LP4LLKPCTYZD7GXSLGARXBEP6LSBGWWMLE75TBWLMP2CXKIA, seg_pos={0 0}, upload=473956]: Done uploading piece. err: <nil>
Dec 28 03:55:25.890982595 (540h26m25.76206564s) [noise=true , address=145.79.227.45:4230 , piece_id=P3YOHIYAXFYL2QEEVSWBHJCNUQNURDIHLJW5RDA3CCE2BTAZNEDQ, seg_pos={0 0}, upload=473956]: Done uploading piece. err: <nil>
Dec 28 03:55:25.891002452 (540h26m25.762085497s) [seg_pos={0 0}, upload=473956]: Segment reached optimal threshold of 13 pieces.
Dec 28 03:55:25.891225377 (540h26m25.762308422s) [noise=true , address=145.79.227.52:4445 , piece_id=BPCT6SAYOOO5UA4RLRFDESPRY4SQEAMVJCCY357CP5MX2LJKOV2A, seg_pos={0 0}, upload=473956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12vXrkhLqCofhR4Fmfqb98gxnqWQF6vMV4q3ATbRvPfcqi82WSo): protocol: expected piece hash; context canceled; EOF
Dec 28 03:55:25.891235486 (540h26m25.762318521s) [noise=true , address=145.79.227.52:4445 , piece_id=BPCT6SAYOOO5UA4RLRFDESPRY4SQEAMVJCCY357CP5MX2LJKOV2A, seg_pos={0 0}, upload=473956]: Long tail cancellation due to optimal threshold
Dec 28 03:55:25.891365414 (540h26m25.762448449s) [noise=true , address=77.90.31.14:4661 , piece_id=NZGSLU5J527AB4Y4CBQR5XTZ7KFDKRR2XZWFMWPUFAA77GI2IJAQ, seg_pos={0 0}, upload=473956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: write tcp 188.166.225.170:38440->77.90.31.14:4661: use of closed network connection; write tcp 188.166.225.170:38440->77.90.31.14:4661: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38440->77.90.31.14:4661: use of closed network connection
Dec 28 03:55:25.891371205 (540h26m25.76245425s) [noise=true , address=77.90.31.14:4661 , piece_id=NZGSLU5J527AB4Y4CBQR5XTZ7KFDKRR2XZWFMWPUFAA77GI2IJAQ, seg_pos={0 0}, upload=473956]: Long tail cancellation due to optimal threshold
Dec 28 03:55:25.891655446 (540h26m25.762738501s) [noise=true , address=145.79.227.16:4909 , piece_id=VBTA33MPR64447LDH6QJBRBRNB2EFIEW5T4O2N73M67F63B66MIQ, seg_pos={0 0}, upload=473956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x): protocol: expected piece hash; context canceled; EOF
Dec 28 03:55:25.891668570 (540h26m25.762751615s) [noise=true , address=145.79.227.16:4909 , piece_id=VBTA33MPR64447LDH6QJBRBRNB2EFIEW5T4O2N73M67F63B66MIQ, seg_pos={0 0}, upload=473956]: Long tail cancellation due to optimal threshold
Dec 28 03:55:25.891693769 (540h26m25.762776814s) [seg_pos={0 0}, upload=473956]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 28 03:55:25.891802816 (540h26m25.762885861s) [upload=473956]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 03:55:25.891810081 (540h26m25.762893126s) [upload=473956]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 03:55:26.194494122 (540h26m26.065577157s) [upload=473956]: Done uploading segments...
Dec 28 03:55:26.194520563 (540h26m26.065603598s) [upload=473956]: Upload finished. err: <nil>
Duration: 30.003189104s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [........................................X...................] duration:
7.185200754s trace:
https://jaeger.dev.storj.io/trace/3b8130f32e60318d
FAIL (2025-12-28 03:55:28 UTC)
orbiter warning: context deadline exceeded
Duration: 7.185223858s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
52 out of 60 is failed (threshold=3) [..XXXXXX.XXXXXXXXXXXXXXXXX.XXXXX.XXXXXXXXXXXXXXXXX..XXXX.XXX] duration:
30.00173742s trace:
https://jaeger.dev.storj.io/trace/5e15639a40e45159
FAIL (2025-12-28 03:55:15 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
ec 28 03:54:46.351219469 (540h25m46.222302504s) [upload=473949]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 28 03:54:46.351223567 (540h25m46.222306602s) [upload=473949]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 28 03:54:46.597019024 (540h25m46.468102059s) [seg_pos={0 0}, upload=473949]: Begin upload segment...
Dec 28 03:54:46.597083085 (540h25m46.46816613s) [seg_pos={0 0}, upload=473949]: Done begin upload segment.
Dec 28 03:54:46.597103955 (540h25m46.468187001s) [noise=true , address=145.79.227.6:4258 , piece_id=5CYCPB5SFCEZBVNF4NABFGIGHBX3DIIUBPNCKDYJYL7AVTRJKV6Q, seg_pos={0 0}, upload=473949]: Uploading piece...
Dec 28 03:54:46.597160844 (540h25m46.468243879s) [noise=true , address=45.91.116.6:4973 , piece_id=U5SMZTVOEOUQVLZGPAXEI2MI7NL6UULPRLZRDZVH2MWYQL3I3KMQ, seg_pos={0 0}, upload=473949]: Uploading piece...
Dec 28 03:54:46.597253239 (540h25m46.468336274s) [noise=true , address=145.79.227.11:4729 , piece_id=TIRESQ52DU2TJUTY7CPHKNXBAVNSDKIRGYCWCJ4Z5LOJBVIJWKFA, seg_pos={0 0}, upload=473949]: Uploading piece...
Dec 28 03:54:46.597719097 (540h25m46.468802132s) [noise=true , address=145.79.227.22:4511 , piece_id=H2Q36DF5UFIBQGPBOTR5EXOFLTC6H7NL62FXVT5B52MRIATXRDEA, seg_pos={0 0}, upload=473949]: Uploading piece...
Dec 28 03:54:46.598014309 (540h25m46.469097354s) [noise=true , address=145.79.227.10:4402 , piece_id=3PX2BP56767RMR6TRKS53W3QWJ7PZZDHNDUBE74Q3PL3QDYPMAPQ, seg_pos={0 0}, upload=473949]: Uploading piece...
Dec 28 03:54:46.598381448 (540h25m46.469464483s) [noise=true , address=45.91.116.7:4010 , piece_id=7U6BVSPMQF3XHYQH2QY23RNQUFXAJ6OAGZWYYXHCLHIEPVJ3QMLQ, seg_pos={0 0}, upload=473949]: Uploading piece...
Dec 28 03:54:46.598840693 (540h25m46.469923728s) [noise=true , address=45.91.116.12:4228 , piece_id=3MJ7CDROFYGQG33HRB7V3MDEPS27ZCC7U7XA4GRX6YEFUWQWBR5Q, seg_pos={0 0}, upload=473949]: Uploading piece...
Dec 28 03:54:46.599225285 (540h25m46.47030832s) [noise=true , address=145.79.227.48:4302 , piece_id=YHTLK2KFSHG7UWG6BF7T3MGN7M2Q4PGESCLEPXFRHDXRQAEXTINA, seg_pos={0 0}, upload=473949]: Uploading piece...
Dec 28 03:54:46.599585981 (540h25m46.470669016s) [noise=true , address=77.90.31.11:4556 , piece_id=RJ3VWZEQR5IAOUER5O3PS6WSTHEINI7MQZ3RUU4ME66QOTQYCJEQ, seg_pos={0 0}, upload=473949]: Uploading piece...
Dec 28 03:54:46.599882837 (540h25m46.470965882s) [noise=true , address=77.90.31.19:4844 , piece_id=E63GRVJ37FXYLDF7ISUIDMISURFCFWFZLNOJPV6IUQUHPQZ2HYCA, seg_pos={0 0}, upload=473949]: Uploading piece...
Dec 28 03:54:46.600938076 (540h25m46.472021111s) [noise=true , address=145.79.227.45:4229 , piece_id=FO3HP7VMIHPICVZ7H3D56EHYVR2AX6L6YBJYVJYJLVBLEXAKCS6A, seg_pos={0 0}, upload=473949]: Uploading piece...
Dec 28 03:54:46.601414192 (540h25m46.472497237s) [noise=true , address=145.79.227.51:4411 , piece_id=7M3QYUTTR4EDV362TWKJWBBJ56IBJS6I4XF5IKHQFVUSVKSEOKXQ, seg_pos={0 0}, upload=473949]: Uploading piece...
Dec 28 03:54:46.601784367 (540h25m46.472867402s) [noise=true , address=45.91.116.14:4189 , piece_id=PIGPLNMOO44IWUFXFX3CCVXCTUDZ6L7I77LBGO2O7C7ASXJLDE2A, seg_pos={0 0}, upload=473949]: Uploading piece...
Dec 28 03:54:46.602247319 (540h25m46.473330354s) [noise=true , address=145.79.227.52:4446 , piece_id=JLZLBMU2EPVUKIC5NAYK5PK24NGSNNFHPAIOFB457ANJVFPYYX5Q, seg_pos={0 0}, upload=473949]: Uploading piece...
Dec 28 03:54:46.602555506 (540h25m46.473638541s) [noise=true , address=77.90.31.15:4697 , piece_id=YSG3TUGPKV2NIC6D2T264JWWH75NTJTO24ZZHCVZ3G6BF76OWZEA, seg_pos={0 0}, upload=473949]: Uploading piece...
Dec 28 03:54:46.603000904 (540h25m46.474083949s) [noise=true , address=145.79.227.32:4835 , piece_id=Y5URNB4QYU2PFCZFB2SL7BJC5VD7ED224WNBOAGI4BPQ4OEPXXTA, seg_pos={0 0}, upload=473949]: Uploading piece...
Dec 28 03:54:48.326027883 (540h25m48.197110928s) [upload=473949]: Next returned nil segment
Dec 28 03:54:48.326040366 (540h25m48.197123411s) [upload=473949]: Waiting for error group managing segments...
Dec 28 03:54:48.614098458 (540h25m48.485181494s) [noise=true , address=145.79.227.32:4835 , piece_id=Y5URNB4QYU2PFCZFB2SL7BJC5VD7ED224WNBOAGI4BPQ4OEPXXTA, seg_pos={0 0}, upload=473949]: Done uploading piece. err: <nil>
Dec 28 03:54:48.629141403 (540h25m48.500224448s) [noise=true , address=145.79.227.52:4446 , piece_id=JLZLBMU2EPVUKIC5NAYK5PK24NGSNNFHPAIOFB457ANJVFPYYX5Q, seg_pos={0 0}, upload=473949]: Done uploading piece. err: <nil>
Dec 28 03:54:48.640682899 (540h25m48.511765934s) [noise=true , address=145.79.227.6:4258 , piece_id=5CYCPB5SFCEZBVNF4NABFGIGHBX3DIIUBPNCKDYJYL7AVTRJKV6Q, seg_pos={0 0}, upload=473949]: Done uploading piece. err: <nil>
Dec 28 03:54:48.790430114 (540h25m48.661513149s) [noise=true , address=77.90.31.15:4697 , piece_id=YSG3TUGPKV2NIC6D2T264JWWH75NTJTO24ZZHCVZ3G6BF76OWZEA, seg_pos={0 0}, upload=473949]: Done uploading piece. err: <nil>
Dec 28 03:54:48.817299338 (540h25m48.688382383s) [noise=true , address=77.90.31.19:4844 , piece_id=E63GRVJ37FXYLDF7ISUIDMISURFCFWFZLNOJPV6IUQUHPQZ2HYCA, seg_pos={0 0}, upload=473949]: Done uploading piece. err: <nil>
Dec 28 03:54:48.849765169 (540h25m48.720848214s) [noise=true , address=45.91.116.14:4189 , piece_id=PIGPLNMOO44IWUFXFX3CCVXCTUDZ6L7I77LBGO2O7C7ASXJLDE2A, seg_pos={0 0}, upload=473949]: Done uploading piece. err: <nil>
Dec 28 03:54:48.880897019 (540h25m48.751980054s) [noise=true , address=45.91.116.6:4973 , piece_id=U5SMZTVOEOUQVLZGPAXEI2MI7NL6UULPRLZRDZVH2MWYQL3I3KMQ, seg_pos={0 0}, upload=473949]: Done uploading piece. err: <nil>
Dec 28 03:54:48.964571158 (540h25m48.835654193s) [noise=true , address=145.79.227.45:4229 , piece_id=FO3HP7VMIHPICVZ7H3D56EHYVR2AX6L6YBJYVJYJLVBLEXAKCS6A, seg_pos={0 0}, upload=473949]: Done uploading piece. err: <nil>
Dec 28 03:54:48.980565054 (540h25m48.851648089s) [noise=true , address=77.90.31.11:4556 , piece_id=RJ3VWZEQR5IAOUER5O3PS6WSTHEINI7MQZ3RUU4ME66QOTQYCJEQ, seg_pos={0 0}, upload=473949]: Done uploading piece. err: <nil>
Dec 28 03:54:48.997281124 (540h25m48.868364169s) [noise=true , address=145.79.227.51:4411 , piece_id=7M3QYUTTR4EDV362TWKJWBBJ56IBJS6I4XF5IKHQFVUSVKSEOKXQ, seg_pos={0 0}, upload=473949]: Done uploading piece. err: <nil>
Dec 28 03:54:49.351145152 (540h25m49.222228197s) [noise=true , address=145.79.227.10:4402 , piece_id=3PX2BP56767RMR6TRKS53W3QWJ7PZZDHNDUBE74Q3PL3QDYPMAPQ, seg_pos={0 0}, upload=473949]: Done uploading piece. err: <nil>
Dec 28 03:54:49.363292332 (540h25m49.234375367s) [noise=true , address=145.79.227.22:4511 , piece_id=H2Q36DF5UFIBQGPBOTR5EXOFLTC6H7NL62FXVT5B52MRIATXRDEA, seg_pos={0 0}, upload=473949]: Done uploading piece. err: <nil>
Dec 28 03:54:49.377389936 (540h25m49.248472971s) [noise=true , address=145.79.227.11:4729 , piece_id=TIRESQ52DU2TJUTY7CPHKNXBAVNSDKIRGYCWCJ4Z5LOJBVIJWKFA, seg_pos={0 0}, upload=473949]: Done uploading piece. err: <nil>
Dec 28 03:54:49.377408292 (540h25m49.248491337s) [seg_pos={0 0}, upload=473949]: Segment reached optimal threshold of 13 pieces.
Dec 28 03:54:49.377584096 (540h25m49.248667142s) [noise=true , address=45.91.116.12:4228 , piece_id=3MJ7CDROFYGQG33HRB7V3MDEPS27ZCC7U7XA4GRX6YEFUWQWBR5Q, seg_pos={0 0}, upload=473949]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: write tcp 188.166.225.170:33710->45.91.116.12:4228: use of closed network connection; write tcp 188.166.225.170:33710->45.91.116.12:4228: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33710->45.91.116.12:4228: use of closed network connection
Dec 28 03:54:49.377590048 (540h25m49.248673083s) [noise=true , address=45.91.116.12:4228 , piece_id=3MJ7CDROFYGQG33HRB7V3MDEPS27ZCC7U7XA4GRX6YEFUWQWBR5Q, seg_pos={0 0}, upload=473949]: Long tail cancellation due to optimal threshold
Dec 28 03:54:49.377677173 (540h25m49.248760208s) [noise=true , address=145.79.227.48:4302 , piece_id=YHTLK2KFSHG7UWG6BF7T3MGN7M2Q4PGESCLEPXFRHDXRQAEXTINA, seg_pos={0 0}, upload=473949]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tGpuYdPLiy4ncDjxrZbQfp7Bq79NVBHDmr4vjaL4qMpudrvgR): protocol: write tcp 188.166.225.170:55826->145.79.227.48:4302: use of closed network connection; write tcp 188.166.225.170:55826->145.79.227.48:4302: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55826->145.79.227.48:4302: use of closed network connection
Dec 28 03:54:49.377682032 (540h25m49.248765067s) [noise=true , address=145.79.227.48:4302 , piece_id=YHTLK2KFSHG7UWG6BF7T3MGN7M2Q4PGESCLEPXFRHDXRQAEXTINA, seg_pos={0 0}, upload=473949]: Long tail cancellation due to optimal threshold
Dec 28 03:54:49.377780149 (540h25m49.248863184s) [noise=true , address=45.91.116.7:4010 , piece_id=7U6BVSPMQF3XHYQH2QY23RNQUFXAJ6OAGZWYYXHCLHIEPVJ3QMLQ, seg_pos={0 0}, upload=473949]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: read tcp 188.166.225.170:34334->45.91.116.7:4010: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 28 03:54:49.377830866 (540h25m49.248913901s) [noise=true , address=45.91.116.7:4010 , piece_id=7U6BVSPMQF3XHYQH2QY23RNQUFXAJ6OAGZWYYXHCLHIEPVJ3QMLQ, seg_pos={0 0}, upload=473949]: Long tail cancellation due to optimal threshold
Dec 28 03:54:49.377854731 (540h25m49.248937766s) [seg_pos={0 0}, upload=473949]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 28 03:54:49.377940725 (540h25m49.24902376s) [upload=473949]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 03:54:49.377945735 (540h25m49.24902877s) [upload=473949]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 03:54:49.671071246 (540h25m49.542154291s) [upload=473949]: Done uploading segments...
Dec 28 03:54:49.671094260 (540h25m49.542177305s) [upload=473949]: Upload finished. err: <nil>
Duration: 30.001748251s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
21 out of 60 is failed (threshold=6) [X.....XX.....X..X.X....X.XXXX....X.X.X...X..X.X.X..X...XX...] duration:
3.700715977s trace:
https://jaeger.dev.storj.io/trace/7e7d6d3ddfeaade0
FAIL (2025-12-28 03:55:11 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.700746948s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [.................X...X......................................] duration:
47.90972554s trace:
https://jaeger.dev.storj.io/trace/54422f75ff432c0
FAIL (2025-12-28 03:54:57 UTC)
orbiter warning:
operation: DeleteObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
, piece_id=SPQSUVHIHI32ZMIRE4NHSNJCMWS4NS6ZZPYF6FV7FQH6VADQIV5A, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:35.720215866 (539h54m35.591298911s) [noise=true , address=145.79.227.43:4187 , piece_id=MRILXYFSWVSH35ZCCFZWLEXENWJII6G6LOR2N277GIBMI4PXPPTQ, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:35.720567665 (539h54m35.5916507s) [noise=true , address=145.79.227.33:4896 , piece_id=7NRM7I4IDHW2WGOCG3B6L6HN74ANXWDJCFRPBM4BCHWJELSNNUXA, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:35.720883607 (539h54m35.591966652s) [noise=true , address=145.79.227.40:4062 , piece_id=RCVKQLDSWIWX2FQBTUFPEUXQZLP2C3ZCLV2JWHOV7CMJ7ZBNWZNQ, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:35.721874904 (539h54m35.592957949s) [noise=true , address=64.49.13.140:4513 , piece_id=Y3HW5I2FX2CWSPVUS7CU3JPNN2RHP2ASL757AUYINBKRKCLXAJGQ, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:35.722249658 (539h54m35.593332693s) [noise=true , address=64.49.13.169:4715 , piece_id=FRVNWFP2AE4DXE4RKZRE6WTK43T74BTTYM2U7E4P3B4XZIMC2QMQ, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:35.722786881 (539h54m35.593869926s) [noise=true , address=77.90.31.14:4673 , piece_id=ZIB5VOFL7PJSRFDV3HP5XAKYHOJXRVHCYDPIHKRYPNM5ROAY5DTQ, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:35.723112100 (539h54m35.594195135s) [noise=true , address=77.90.31.17:4779 , piece_id=TLU5ZLZRVAMGQLSE2LLNIUE4WOWJBMFJGFQKC3KOUJ23V3UVFMRA, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:35.723440756 (539h54m35.594523791s) [noise=true , address=145.79.227.37:4569 , piece_id=DUXSQUT36XUF66ZVQ2V7ZWNXA6JZKBDO5RXT3QVT26GE2FTH3QXQ, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:35.723849544 (539h54m35.594932589s) [noise=true , address=145.79.227.27:4723 , piece_id=4DECSGDNZLXNQVPU7QALLPFQYDO7MOG6TIBNWQ5UHNVCIHB4GUYQ, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:35.724241290 (539h54m35.595324335s) [noise=true , address=77.90.31.11:4587 , piece_id=6467VZCLWHJTZ36ENMEOEOU2E4MAUCDL2XHNAOMJTCCTUSN33NFA, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:35.724607808 (539h54m35.595690854s) [noise=true , address=145.79.227.47:4272 , piece_id=FL2U4W2YCV5ANDVFHB7H4TFT3SYTBGNLI3KZDI7MY2K6NTSMRUAA, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:35.724974035 (539h54m35.59605707s) [noise=true , address=145.79.227.18:4040 , piece_id=W55GK3QDHRHNR6YJ7JKVZMIEKFQLZSTXKSQ5OMW3U7XRI44EM6RA, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:35.725302281 (539h54m35.596385316s) [noise=true , address=145.79.227.38:4989 , piece_id=OUINZ6BXOLFCDNCXMX7PPFSKWMJ7IPZWO7DAGLGVV2AB7UHO3JYQ, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:35.726824420 (539h54m35.597907455s) [noise=true , address=145.79.227.26:4669 , piece_id=5D7SGVO274FEZ4JVESTMUYQ3RZ3DJA5DL2QZV5YKMXEB4TM6PHAA, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:37.063163473 (539h54m36.934246518s) [upload=473490]: Next returned nil segment
Dec 28 03:23:37.063176499 (539h54m36.934259534s) [upload=473490]: Waiting for error group managing segments...
Dec 28 03:23:37.416516311 (539h54m37.287599376s) [noise=true , address=64.49.13.140:4513 , piece_id=Y3HW5I2FX2CWSPVUS7CU3JPNN2RHP2ASL757AUYINBKRKCLXAJGQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil>
Dec 28 03:23:37.606629586 (539h54m37.477712622s) [noise=true , address=145.79.227.27:4723 , piece_id=4DECSGDNZLXNQVPU7QALLPFQYDO7MOG6TIBNWQ5UHNVCIHB4GUYQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil>
Dec 28 03:23:37.617730204 (539h54m37.488813249s) [noise=true , address=145.79.227.26:4669 , piece_id=5D7SGVO274FEZ4JVESTMUYQ3RZ3DJA5DL2QZV5YKMXEB4TM6PHAA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil>
Dec 28 03:23:37.632542950 (539h54m37.503625995s) [noise=true , address=145.79.227.38:4989 , piece_id=OUINZ6BXOLFCDNCXMX7PPFSKWMJ7IPZWO7DAGLGVV2AB7UHO3JYQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil>
Dec 28 03:23:37.643987531 (539h54m37.515070566s) [noise=true , address=145.79.227.47:4272 , piece_id=FL2U4W2YCV5ANDVFHB7H4TFT3SYTBGNLI3KZDI7MY2K6NTSMRUAA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil>
Dec 28 03:23:37.652913619 (539h54m37.523996654s) [noise=true , address=145.79.227.18:4040 , piece_id=W55GK3QDHRHNR6YJ7JKVZMIEKFQLZSTXKSQ5OMW3U7XRI44EM6RA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil>
Dec 28 03:23:37.661966357 (539h54m37.533049392s) [noise=true , address=145.79.227.37:4569 , piece_id=DUXSQUT36XUF66ZVQ2V7ZWNXA6JZKBDO5RXT3QVT26GE2FTH3QXQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil>
Dec 28 03:23:37.778627848 (539h54m37.649710893s) [noise=true , address=77.90.31.11:4561 , piece_id=HIUCZRRZOMU4RBYZU5SADJAOS4RVML2I6PILIX2V65RS22EBN42Q, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil>
Dec 28 03:23:37.815974812 (539h54m37.687057857s) [noise=true , address=45.91.116.14:4217 , piece_id=SPQSUVHIHI32ZMIRE4NHSNJCMWS4NS6ZZPYF6FV7FQH6VADQIV5A, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil>
Dec 28 03:23:37.861731780 (539h54m37.732814815s) [noise=true , address=77.90.31.14:4673 , piece_id=ZIB5VOFL7PJSRFDV3HP5XAKYHOJXRVHCYDPIHKRYPNM5ROAY5DTQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil>
Dec 28 03:23:38.093273211 (539h54m37.964356256s) [noise=true , address=77.90.31.11:4587 , piece_id=6467VZCLWHJTZ36ENMEOEOU2E4MAUCDL2XHNAOMJTCCTUSN33NFA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil>
Dec 28 03:23:38.267694395 (539h54m38.13877745s) [noise=true , address=77.90.31.17:4779 , piece_id=TLU5ZLZRVAMGQLSE2LLNIUE4WOWJBMFJGFQKC3KOUJ23V3UVFMRA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil>
Dec 28 03:23:38.540947761 (539h54m38.412030796s) [noise=true , address=145.79.227.40:4062 , piece_id=RCVKQLDSWIWX2FQBTUFPEUXQZLP2C3ZCLV2JWHOV7CMJ7ZBNWZNQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil>
Dec 28 03:23:38.815393527 (539h54m38.686476562s) [noise=true , address=145.79.227.43:4187 , piece_id=MRILXYFSWVSH35ZCCFZWLEXENWJII6G6LOR2N277GIBMI4PXPPTQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil>
Dec 28 03:23:38.815412894 (539h54m38.686495939s) [seg_pos={0 0}, upload=473490]: Segment reached optimal threshold of 14 pieces.
Dec 28 03:23:38.815594951 (539h54m38.686677986s) [noise=true , address=64.49.13.169:4715 , piece_id=FRVNWFP2AE4DXE4RKZRE6WTK43T74BTTYM2U7E4P3B4XZIMC2QMQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:43502->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:43502->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43502->64.49.13.169:4715: use of closed network connection
Dec 28 03:23:38.815601032 (539h54m38.686684077s) [noise=true , address=64.49.13.169:4715 , piece_id=FRVNWFP2AE4DXE4RKZRE6WTK43T74BTTYM2U7E4P3B4XZIMC2QMQ, seg_pos={0 0}, upload=473490]: Long tail cancellation due to optimal threshold
Dec 28 03:23:38.815826983 (539h54m38.686910018s) [noise=true , address=145.79.227.41:4118 , piece_id=C7JJ3NHR5355M64G2EF2WWERQVGWR243RQHDV3GBAIC3A66JDMHQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128w37UHxJB9R49iwmHbATt3bXUb1FMTQJ3Wjka7So7eqKWg4iF): protocol: expected piece hash; context canceled; EOF
Dec 28 03:23:38.815834507 (539h54m38.686917542s) [noise=true , address=145.79.227.41:4118 , piece_id=C7JJ3NHR5355M64G2EF2WWERQVGWR243RQHDV3GBAIC3A66JDMHQ, seg_pos={0 0}, upload=473490]: Long tail cancellation due to optimal threshold
Dec 28 03:23:38.815890433 (539h54m38.686973468s) [noise=true , address=145.79.227.33:4896 , piece_id=7NRM7I4IDHW2WGOCG3B6L6HN74ANXWDJCFRPBM4BCHWJELSNNUXA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ndSgjgfZMVQni5tw1ugdTrhJt4c4NagdvgYTbSbjatYYtfu1o): protocol: expected piece hash; context canceled; EOF
Dec 28 03:23:38.815914619 (539h54m38.686997654s) [noise=true , address=145.79.227.33:4896 , piece_id=7NRM7I4IDHW2WGOCG3B6L6HN74ANXWDJCFRPBM4BCHWJELSNNUXA, seg_pos={0 0}, upload=473490]: Long tail cancellation due to optimal threshold
Dec 28 03:23:38.816105613 (539h54m38.687188648s) [noise=true , address=145.79.227.28:4731 , piece_id=EPW3LUA3EWRCURHZ66NCRCA36CNIW4GYQMFNKPZZTRCB3YBGTKJA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NQzTwVYQa6ZzxAHQm1apkwWnqJstTnAJQyHKhBzC2jTN9kEVK): protocol: write tcp 188.166.225.170:58940->145.79.227.28:4731: use of closed network connection; write tcp 188.166.225.170:58940->145.79.227.28:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58940->145.79.227.28:4731: use of closed network connection
Dec 28 03:23:38.816112426 (539h54m38.687195461s) [noise=true , address=145.79.227.28:4731 , piece_id=EPW3LUA3EWRCURHZ66NCRCA36CNIW4GYQMFNKPZZTRCB3YBGTKJA, seg_pos={0 0}, upload=473490]: Long tail cancellation due to optimal threshold
Dec 28 03:23:38.816128436 (539h54m38.687211481s) [seg_pos={0 0}, upload=473490]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 28 03:23:38.816257582 (539h54m38.687340627s) [upload=473490]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 03:23:38.816265778 (539h54m38.687348813s) [upload=473490]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 03:23:39.111115407 (539h54m38.982198442s) [upload=473490]: Done uploading segments...
Dec 28 03:23:39.111138972 (539h54m38.982222008s) [upload=473490]: Upload finished. err: <nil>
Duration: 47.909739977s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.........................................X..................] duration:
6.87208181s trace:
https://jaeger.dev.storj.io/trace/3b8130f32e60318d
FAIL (2025-12-28 03:54:47 UTC)
orbiter warning: context deadline exceeded
Duration: 6.872105296s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
52 out of 60 is failed (threshold=3) [X..XXXXXX.XXXXXXXXXXXXXXXXX.XXXXX.XXXXXXXXXXXXXXXXX..XXXX.XX] duration:
30.003587023s trace:
https://jaeger.dev.storj.io/trace/55a25ce68a7af12d
FAIL (2025-12-28 03:54:45 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 28 03:54:16.407404993 (540h25m16.278488028s) [upload=473942]: Starting upload
Dec 28 03:54:16.407488822 (540h25m16.278571878s) [upload=473942]: Done starting upload
Dec 28 03:54:16.407629430 (540h25m16.278712475s) [upload=473942]: Uploading segments...
Dec 28 03:54:16.407834861 (540h25m16.278917906s) [upload=473942]: Got next segment. Inline: false
Dec 28 03:54:16.407892672 (540h25m16.278975707s) [upload=473942]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 28 03:54:16.407897511 (540h25m16.278980546s) [upload=473942]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 28 03:54:16.758921771 (540h25m16.630004816s) [seg_pos={0 0}, upload=473942]: Begin upload segment...
Dec 28 03:54:16.758982006 (540h25m16.630065051s) [seg_pos={0 0}, upload=473942]: Done begin upload segment.
Dec 28 03:54:16.759003257 (540h25m16.630086302s) [noise=true , address=145.79.227.13:4802 , piece_id=X5GCN2NPRML6PTY2DSIXEUQFHPPUDJNOTSAGTJ3YFACGMGGSYUEQ, seg_pos={0 0}, upload=473942]: Uploading piece...
Dec 28 03:54:16.759040808 (540h25m16.630123843s) [noise=true , address=45.91.116.14:4189 , piece_id=RGXKV4KVS3ZHGAUNJMA6DHGI7ABTAV45M4YOZSVBMNSTON2ARDYQ, seg_pos={0 0}, upload=473942]: Uploading piece...
Dec 28 03:54:16.759210211 (540h25m16.630293246s) [noise=true , address=45.91.116.10:4084 , piece_id=G2ANA77Y3OIDPTB5IC4PNTK3WQ5V6AZENOY7ISTOFH2MYA2B5QDQ, seg_pos={0 0}, upload=473942]: Uploading piece...
Dec 28 03:54:16.759869946 (540h25m16.630952981s) [noise=true , address=145.79.227.48:4301 , piece_id=SLCYNBW3FZERUE5JPK3YV4UQ5XO4BRRK6BCENG2UF7OW6LL6E24Q, seg_pos={0 0}, upload=473942]: Uploading piece...
Dec 28 03:54:16.760289696 (540h25m16.631372741s) [noise=true , address=145.79.227.24:4583 , piece_id=UILNAPRP6ZSY7E5UYV7MOJQYF3NRCK24RG55E64FS3VQFD4MZ5XQ, seg_pos={0 0}, upload=473942]: Uploading piece...
Dec 28 03:54:16.760723101 (540h25m16.631806146s) [noise=true , address=45.91.116.9:4262 , piece_id=7MTY4RLANTJX7KGZLMYNZDFVFSL2ZR5PDUFRZ32KTMH34WM3B6WQ, seg_pos={0 0}, upload=473942]: Uploading piece...
Dec 28 03:54:16.761276445 (540h25m16.63235949s) [noise=true , address=145.79.227.32:4835 , piece_id=WFGU5HDFUGG42DOKVLBTWKMIWGU54KNM525LNCTAELBCGUAEBBZA, seg_pos={0 0}, upload=473942]: Uploading piece...
Dec 28 03:54:16.761732834 (540h25m16.632815869s) [noise=true , address=77.90.31.14:4662 , piece_id=75ZDCMO24YHII37WSKC7VAU27PQSNN2VWWGLIFLHPR43YK62JSQA, seg_pos={0 0}, upload=473942]: Uploading piece...
Dec 28 03:54:16.762324020 (540h25m16.633407055s) [noise=true , address=77.90.31.12:4589 , piece_id=VDQ5HW2SNVSC5V3WS6I5IITYCNOSSNDLFALVF25SO3UEJLFQDK7A, seg_pos={0 0}, upload=473942]: Uploading piece...
Dec 28 03:54:16.763238200 (540h25m16.634321255s) [noise=true , address=145.79.227.4:4186 , piece_id=U7HLNIM5B754YEA5GABQKFZROXB2DVR5SNPWMBZDYXBYVJ2UWHPA, seg_pos={0 0}, upload=473942]: Uploading piece...
Dec 28 03:54:16.765217790 (540h25m16.636300825s) [noise=true , address=145.79.227.38:4979 , piece_id=VNDELBUYOZBVHR4TPJLHWYJ3W5ORAHZ73ARLNHD5PJLZP4JTLGYA, seg_pos={0 0}, upload=473942]: Uploading piece...
Dec 28 03:54:16.765252766 (540h25m16.636335801s) [noise=true , address=45.91.116.13:4156 , piece_id=NYNA6DJCRSR5DGSRYEO4WJNM6QADMR66MEJ7I727ZCFKYP5SFB6Q, seg_pos={0 0}, upload=473942]: Uploading piece...
Dec 28 03:54:16.767319703 (540h25m16.638402738s) [noise=true , address=145.79.227.15:4874 , piece_id=D36HMVJQNF2M4XLHEQBL7C55KJDBTJLTTTWVAGDCK3YQD255ZXSQ, seg_pos={0 0}, upload=473942]: Uploading piece...
Dec 28 03:54:16.770535035 (540h25m16.64161809s) [noise=true , address=145.79.227.5:4221 , piece_id=6JKXVFYYLY6LUQ6H44ZMYQUILDC45VTHZGS6AYVKNOZTULQEOV6A, seg_pos={0 0}, upload=473942]: Uploading piece...
Dec 28 03:54:16.773712315 (540h25m16.64479536s) [noise=true , address=77.90.31.10:4518 , piece_id=7ASEWFSSBHCTET5KFDCMZGNQKU6JGBL5CK7FFAMKKCT377ARUJ7Q, seg_pos={0 0}, upload=473942]: Uploading piece...
Dec 28 03:54:16.776365717 (540h25m16.647448762s) [noise=true , address=145.79.227.16:4912 , piece_id=RDKYRYCKRQKFHUIMSLOTGVNZWJ6XUJFJQM56BSWCNDBLR6I6X4GQ, seg_pos={0 0}, upload=473942]: Uploading piece...
Dec 28 03:54:18.474042705 (540h25m18.34512575s) [upload=473942]: Next returned nil segment
Dec 28 03:54:18.474059457 (540h25m18.345142502s) [upload=473942]: Waiting for error group managing segments...
Dec 28 03:54:18.783561963 (540h25m18.654644998s) [noise=true , address=145.79.227.24:4583 , piece_id=UILNAPRP6ZSY7E5UYV7MOJQYF3NRCK24RG55E64FS3VQFD4MZ5XQ, seg_pos={0 0}, upload=473942]: Done uploading piece. err: <nil>
Dec 28 03:54:18.784583047 (540h25m18.655666122s) [noise=true , address=145.79.227.16:4912 , piece_id=RDKYRYCKRQKFHUIMSLOTGVNZWJ6XUJFJQM56BSWCNDBLR6I6X4GQ, seg_pos={0 0}, upload=473942]: Done uploading piece. err: <nil>
Dec 28 03:54:18.793664329 (540h25m18.664747364s) [noise=true , address=145.79.227.15:4874 , piece_id=D36HMVJQNF2M4XLHEQBL7C55KJDBTJLTTTWVAGDCK3YQD255ZXSQ, seg_pos={0 0}, upload=473942]: Done uploading piece. err: <nil>
Dec 28 03:54:18.797782442 (540h25m18.668865487s) [noise=true , address=145.79.227.48:4301 , piece_id=SLCYNBW3FZERUE5JPK3YV4UQ5XO4BRRK6BCENG2UF7OW6LL6E24Q, seg_pos={0 0}, upload=473942]: Done uploading piece. err: <nil>
Dec 28 03:54:18.809449367 (540h25m18.680532412s) [noise=true , address=145.79.227.13:4802 , piece_id=X5GCN2NPRML6PTY2DSIXEUQFHPPUDJNOTSAGTJ3YFACGMGGSYUEQ, seg_pos={0 0}, upload=473942]: Done uploading piece. err: <nil>
Dec 28 03:54:18.809908742 (540h25m18.680991777s) [noise=true , address=145.79.227.32:4835 , piece_id=WFGU5HDFUGG42DOKVLBTWKMIWGU54KNM525LNCTAELBCGUAEBBZA, seg_pos={0 0}, upload=473942]: Done uploading piece. err: <nil>
Dec 28 03:54:18.817569309 (540h25m18.688652354s) [noise=true , address=145.79.227.38:4979 , piece_id=VNDELBUYOZBVHR4TPJLHWYJ3W5ORAHZ73ARLNHD5PJLZP4JTLGYA, seg_pos={0 0}, upload=473942]: Done uploading piece. err: <nil>
Dec 28 03:54:18.895832225 (540h25m18.76691528s) [noise=true , address=145.79.227.5:4221 , piece_id=6JKXVFYYLY6LUQ6H44ZMYQUILDC45VTHZGS6AYVKNOZTULQEOV6A, seg_pos={0 0}, upload=473942]: Done uploading piece. err: <nil>
Dec 28 03:54:18.966939205 (540h25m18.83802225s) [noise=true , address=77.90.31.14:4662 , piece_id=75ZDCMO24YHII37WSKC7VAU27PQSNN2VWWGLIFLHPR43YK62JSQA, seg_pos={0 0}, upload=473942]: Done uploading piece. err: <nil>
Dec 28 03:54:19.028127037 (540h25m18.899210102s) [noise=true , address=45.91.116.14:4189 , piece_id=RGXKV4KVS3ZHGAUNJMA6DHGI7ABTAV45M4YOZSVBMNSTON2ARDYQ, seg_pos={0 0}, upload=473942]: Done uploading piece. err: <nil>
Dec 28 03:54:19.126276731 (540h25m18.997359766s) [noise=true , address=45.91.116.13:4156 , piece_id=NYNA6DJCRSR5DGSRYEO4WJNM6QADMR66MEJ7I727ZCFKYP5SFB6Q, seg_pos={0 0}, upload=473942]: Done uploading piece. err: <nil>
Dec 28 03:54:19.170175231 (540h25m19.041258266s) [noise=true , address=45.91.116.10:4084 , piece_id=G2ANA77Y3OIDPTB5IC4PNTK3WQ5V6AZENOY7ISTOFH2MYA2B5QDQ, seg_pos={0 0}, upload=473942]: Done uploading piece. err: <nil>
Dec 28 03:54:19.800197100 (540h25m19.671280135s) [noise=true , address=45.91.116.9:4262 , piece_id=7MTY4RLANTJX7KGZLMYNZDFVFSL2ZR5PDUFRZ32KTMH34WM3B6WQ, seg_pos={0 0}, upload=473942]: Done uploading piece. err: <nil>
Dec 28 03:54:19.800215025 (540h25m19.67129806s) [seg_pos={0 0}, upload=473942]: Segment reached optimal threshold of 13 pieces.
Dec 28 03:54:19.800373456 (540h25m19.671456491s) [noise=true , address=145.79.227.4:4186 , piece_id=U7HLNIM5B754YEA5GABQKFZROXB2DVR5SNPWMBZDYXBYVJ2UWHPA, seg_pos={0 0}, upload=473942]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht): protocol: expected piece hash; context canceled; EOF
Dec 28 03:54:19.800379649 (540h25m19.671462684s) [noise=true , address=145.79.227.4:4186 , piece_id=U7HLNIM5B754YEA5GABQKFZROXB2DVR5SNPWMBZDYXBYVJ2UWHPA, seg_pos={0 0}, upload=473942]: Long tail cancellation due to optimal threshold
Dec 28 03:54:19.800525426 (540h25m19.671608461s) [noise=true , address=77.90.31.10:4518 , piece_id=7ASEWFSSBHCTET5KFDCMZGNQKU6JGBL5CK7FFAMKKCT377ARUJ7Q, seg_pos={0 0}, upload=473942]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF
Dec 28 03:54:19.800533872 (540h25m19.671616907s) [noise=true , address=77.90.31.10:4518 , piece_id=7ASEWFSSBHCTET5KFDCMZGNQKU6JGBL5CK7FFAMKKCT377ARUJ7Q, seg_pos={0 0}, upload=473942]: Long tail cancellation due to optimal threshold
Dec 28 03:54:19.800620708 (540h25m19.671703763s) [noise=true , address=77.90.31.12:4589 , piece_id=VDQ5HW2SNVSC5V3WS6I5IITYCNOSSNDLFALVF25SO3UEJLFQDK7A, seg_pos={0 0}, upload=473942]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF
Dec 28 03:54:19.800643200 (540h25m19.671726235s) [noise=true , address=77.90.31.12:4589 , piece_id=VDQ5HW2SNVSC5V3WS6I5IITYCNOSSNDLFALVF25SO3UEJLFQDK7A, seg_pos={0 0}, upload=473942]: Long tail cancellation due to optimal threshold
Dec 28 03:54:19.800699317 (540h25m19.671782352s) [seg_pos={0 0}, upload=473942]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 28 03:54:19.800821068 (540h25m19.671904113s) [upload=473942]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 03:54:19.800829195 (540h25m19.67191225s) [upload=473942]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 03:54:20.174669179 (540h25m20.045752234s) [upload=473942]: Done uploading segments...
Dec 28 03:54:20.174696832 (540h25m20.045779877s) [upload=473942]: Upload finished. err: <nil>
Duration: 30.003687265s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..........................................X.................] duration:
15.073960426s trace:
https://jaeger.dev.storj.io/trace/3b8130f32e60318d
FAIL (2025-12-28 03:54:33 UTC)
orbiter warning: context deadline exceeded
Duration: 15.073985272s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
21 out of 60 is failed (threshold=6) [.X.....XX.....X..X.X....X.XXXX....X.X.X...X..X.X.X..X...XX..] duration:
3.461235888s trace:
https://jaeger.dev.storj.io/trace/7e7d6d3ddfeaade0
FAIL (2025-12-28 03:54:29 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.461265263s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [..................X...X.....................................] duration:
42.012778936s trace:
https://jaeger.dev.storj.io/trace/54422f75ff432c0
FAIL (2025-12-28 03:54:09 UTC)
orbiter warning:
operation: DeleteObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
, piece_id=SPQSUVHIHI32ZMIRE4NHSNJCMWS4NS6ZZPYF6FV7FQH6VADQIV5A, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:35.720215866 (539h54m35.591298911s) [noise=true , address=145.79.227.43:4187 , piece_id=MRILXYFSWVSH35ZCCFZWLEXENWJII6G6LOR2N277GIBMI4PXPPTQ, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:35.720567665 (539h54m35.5916507s) [noise=true , address=145.79.227.33:4896 , piece_id=7NRM7I4IDHW2WGOCG3B6L6HN74ANXWDJCFRPBM4BCHWJELSNNUXA, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:35.720883607 (539h54m35.591966652s) [noise=true , address=145.79.227.40:4062 , piece_id=RCVKQLDSWIWX2FQBTUFPEUXQZLP2C3ZCLV2JWHOV7CMJ7ZBNWZNQ, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:35.721874904 (539h54m35.592957949s) [noise=true , address=64.49.13.140:4513 , piece_id=Y3HW5I2FX2CWSPVUS7CU3JPNN2RHP2ASL757AUYINBKRKCLXAJGQ, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:35.722249658 (539h54m35.593332693s) [noise=true , address=64.49.13.169:4715 , piece_id=FRVNWFP2AE4DXE4RKZRE6WTK43T74BTTYM2U7E4P3B4XZIMC2QMQ, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:35.722786881 (539h54m35.593869926s) [noise=true , address=77.90.31.14:4673 , piece_id=ZIB5VOFL7PJSRFDV3HP5XAKYHOJXRVHCYDPIHKRYPNM5ROAY5DTQ, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:35.723112100 (539h54m35.594195135s) [noise=true , address=77.90.31.17:4779 , piece_id=TLU5ZLZRVAMGQLSE2LLNIUE4WOWJBMFJGFQKC3KOUJ23V3UVFMRA, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:35.723440756 (539h54m35.594523791s) [noise=true , address=145.79.227.37:4569 , piece_id=DUXSQUT36XUF66ZVQ2V7ZWNXA6JZKBDO5RXT3QVT26GE2FTH3QXQ, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:35.723849544 (539h54m35.594932589s) [noise=true , address=145.79.227.27:4723 , piece_id=4DECSGDNZLXNQVPU7QALLPFQYDO7MOG6TIBNWQ5UHNVCIHB4GUYQ, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:35.724241290 (539h54m35.595324335s) [noise=true , address=77.90.31.11:4587 , piece_id=6467VZCLWHJTZ36ENMEOEOU2E4MAUCDL2XHNAOMJTCCTUSN33NFA, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:35.724607808 (539h54m35.595690854s) [noise=true , address=145.79.227.47:4272 , piece_id=FL2U4W2YCV5ANDVFHB7H4TFT3SYTBGNLI3KZDI7MY2K6NTSMRUAA, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:35.724974035 (539h54m35.59605707s) [noise=true , address=145.79.227.18:4040 , piece_id=W55GK3QDHRHNR6YJ7JKVZMIEKFQLZSTXKSQ5OMW3U7XRI44EM6RA, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:35.725302281 (539h54m35.596385316s) [noise=true , address=145.79.227.38:4989 , piece_id=OUINZ6BXOLFCDNCXMX7PPFSKWMJ7IPZWO7DAGLGVV2AB7UHO3JYQ, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:35.726824420 (539h54m35.597907455s) [noise=true , address=145.79.227.26:4669 , piece_id=5D7SGVO274FEZ4JVESTMUYQ3RZ3DJA5DL2QZV5YKMXEB4TM6PHAA, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:37.063163473 (539h54m36.934246518s) [upload=473490]: Next returned nil segment
Dec 28 03:23:37.063176499 (539h54m36.934259534s) [upload=473490]: Waiting for error group managing segments...
Dec 28 03:23:37.416516311 (539h54m37.287599376s) [noise=true , address=64.49.13.140:4513 , piece_id=Y3HW5I2FX2CWSPVUS7CU3JPNN2RHP2ASL757AUYINBKRKCLXAJGQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil>
Dec 28 03:23:37.606629586 (539h54m37.477712622s) [noise=true , address=145.79.227.27:4723 , piece_id=4DECSGDNZLXNQVPU7QALLPFQYDO7MOG6TIBNWQ5UHNVCIHB4GUYQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil>
Dec 28 03:23:37.617730204 (539h54m37.488813249s) [noise=true , address=145.79.227.26:4669 , piece_id=5D7SGVO274FEZ4JVESTMUYQ3RZ3DJA5DL2QZV5YKMXEB4TM6PHAA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil>
Dec 28 03:23:37.632542950 (539h54m37.503625995s) [noise=true , address=145.79.227.38:4989 , piece_id=OUINZ6BXOLFCDNCXMX7PPFSKWMJ7IPZWO7DAGLGVV2AB7UHO3JYQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil>
Dec 28 03:23:37.643987531 (539h54m37.515070566s) [noise=true , address=145.79.227.47:4272 , piece_id=FL2U4W2YCV5ANDVFHB7H4TFT3SYTBGNLI3KZDI7MY2K6NTSMRUAA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil>
Dec 28 03:23:37.652913619 (539h54m37.523996654s) [noise=true , address=145.79.227.18:4040 , piece_id=W55GK3QDHRHNR6YJ7JKVZMIEKFQLZSTXKSQ5OMW3U7XRI44EM6RA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil>
Dec 28 03:23:37.661966357 (539h54m37.533049392s) [noise=true , address=145.79.227.37:4569 , piece_id=DUXSQUT36XUF66ZVQ2V7ZWNXA6JZKBDO5RXT3QVT26GE2FTH3QXQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil>
Dec 28 03:23:37.778627848 (539h54m37.649710893s) [noise=true , address=77.90.31.11:4561 , piece_id=HIUCZRRZOMU4RBYZU5SADJAOS4RVML2I6PILIX2V65RS22EBN42Q, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil>
Dec 28 03:23:37.815974812 (539h54m37.687057857s) [noise=true , address=45.91.116.14:4217 , piece_id=SPQSUVHIHI32ZMIRE4NHSNJCMWS4NS6ZZPYF6FV7FQH6VADQIV5A, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil>
Dec 28 03:23:37.861731780 (539h54m37.732814815s) [noise=true , address=77.90.31.14:4673 , piece_id=ZIB5VOFL7PJSRFDV3HP5XAKYHOJXRVHCYDPIHKRYPNM5ROAY5DTQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil>
Dec 28 03:23:38.093273211 (539h54m37.964356256s) [noise=true , address=77.90.31.11:4587 , piece_id=6467VZCLWHJTZ36ENMEOEOU2E4MAUCDL2XHNAOMJTCCTUSN33NFA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil>
Dec 28 03:23:38.267694395 (539h54m38.13877745s) [noise=true , address=77.90.31.17:4779 , piece_id=TLU5ZLZRVAMGQLSE2LLNIUE4WOWJBMFJGFQKC3KOUJ23V3UVFMRA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil>
Dec 28 03:23:38.540947761 (539h54m38.412030796s) [noise=true , address=145.79.227.40:4062 , piece_id=RCVKQLDSWIWX2FQBTUFPEUXQZLP2C3ZCLV2JWHOV7CMJ7ZBNWZNQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil>
Dec 28 03:23:38.815393527 (539h54m38.686476562s) [noise=true , address=145.79.227.43:4187 , piece_id=MRILXYFSWVSH35ZCCFZWLEXENWJII6G6LOR2N277GIBMI4PXPPTQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil>
Dec 28 03:23:38.815412894 (539h54m38.686495939s) [seg_pos={0 0}, upload=473490]: Segment reached optimal threshold of 14 pieces.
Dec 28 03:23:38.815594951 (539h54m38.686677986s) [noise=true , address=64.49.13.169:4715 , piece_id=FRVNWFP2AE4DXE4RKZRE6WTK43T74BTTYM2U7E4P3B4XZIMC2QMQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:43502->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:43502->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43502->64.49.13.169:4715: use of closed network connection
Dec 28 03:23:38.815601032 (539h54m38.686684077s) [noise=true , address=64.49.13.169:4715 , piece_id=FRVNWFP2AE4DXE4RKZRE6WTK43T74BTTYM2U7E4P3B4XZIMC2QMQ, seg_pos={0 0}, upload=473490]: Long tail cancellation due to optimal threshold
Dec 28 03:23:38.815826983 (539h54m38.686910018s) [noise=true , address=145.79.227.41:4118 , piece_id=C7JJ3NHR5355M64G2EF2WWERQVGWR243RQHDV3GBAIC3A66JDMHQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128w37UHxJB9R49iwmHbATt3bXUb1FMTQJ3Wjka7So7eqKWg4iF): protocol: expected piece hash; context canceled; EOF
Dec 28 03:23:38.815834507 (539h54m38.686917542s) [noise=true , address=145.79.227.41:4118 , piece_id=C7JJ3NHR5355M64G2EF2WWERQVGWR243RQHDV3GBAIC3A66JDMHQ, seg_pos={0 0}, upload=473490]: Long tail cancellation due to optimal threshold
Dec 28 03:23:38.815890433 (539h54m38.686973468s) [noise=true , address=145.79.227.33:4896 , piece_id=7NRM7I4IDHW2WGOCG3B6L6HN74ANXWDJCFRPBM4BCHWJELSNNUXA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ndSgjgfZMVQni5tw1ugdTrhJt4c4NagdvgYTbSbjatYYtfu1o): protocol: expected piece hash; context canceled; EOF
Dec 28 03:23:38.815914619 (539h54m38.686997654s) [noise=true , address=145.79.227.33:4896 , piece_id=7NRM7I4IDHW2WGOCG3B6L6HN74ANXWDJCFRPBM4BCHWJELSNNUXA, seg_pos={0 0}, upload=473490]: Long tail cancellation due to optimal threshold
Dec 28 03:23:38.816105613 (539h54m38.687188648s) [noise=true , address=145.79.227.28:4731 , piece_id=EPW3LUA3EWRCURHZ66NCRCA36CNIW4GYQMFNKPZZTRCB3YBGTKJA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NQzTwVYQa6ZzxAHQm1apkwWnqJstTnAJQyHKhBzC2jTN9kEVK): protocol: write tcp 188.166.225.170:58940->145.79.227.28:4731: use of closed network connection; write tcp 188.166.225.170:58940->145.79.227.28:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58940->145.79.227.28:4731: use of closed network connection
Dec 28 03:23:38.816112426 (539h54m38.687195461s) [noise=true , address=145.79.227.28:4731 , piece_id=EPW3LUA3EWRCURHZ66NCRCA36CNIW4GYQMFNKPZZTRCB3YBGTKJA, seg_pos={0 0}, upload=473490]: Long tail cancellation due to optimal threshold
Dec 28 03:23:38.816128436 (539h54m38.687211481s) [seg_pos={0 0}, upload=473490]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 28 03:23:38.816257582 (539h54m38.687340627s) [upload=473490]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 03:23:38.816265778 (539h54m38.687348813s) [upload=473490]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 03:23:39.111115407 (539h54m38.982198442s) [upload=473490]: Done uploading segments...
Dec 28 03:23:39.111138972 (539h54m38.982222008s) [upload=473490]: Upload finished. err: <nil>
Duration: 42.012795286s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
21 out of 60 is failed (threshold=6) [..X.....XX.....X..X.X....X.XXXX....X.X.X...X..X.X.X..X...XX.] duration:
3.507929098s trace:
https://jaeger.dev.storj.io/trace/7e7d6d3ddfeaade0
FAIL (2025-12-28 03:53:48 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.507972812s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........................................X................] duration:
13.556959616s trace:
https://jaeger.dev.storj.io/trace/3b8130f32e60318d
FAIL (2025-12-28 03:53:42 UTC)
orbiter warning: context deadline exceeded
Duration: 13.556991708s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
52 out of 60 is failed (threshold=3) [XX..XXXXXX.XXXXXXXXXXXXXXXXX.XXXXX.XXXXXXXXXXXXXXXXX..XXXX.X] duration:
30.003575072s trace:
https://jaeger.dev.storj.io/trace/55d9fa3852578fad
FAIL (2025-12-28 03:53:33 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: XSDJIS4KA72F3676LHIQKKXU3AQB5TER3ILEKOQVIGSCELH6CE6A) context deadline exceeded; piecestore: (Node ID: 12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9, Piece ID: EBANH3SP7FG5YNVQCTALZFSI3JZMEK3UKEGQEWAEG4PZVCFHOIQQ) context deadline exceeded; piecestore: (Node ID: 121fzX9v4rgEd78xnDAwcQtthyQHV3yUZUQYLXZ9byPDucGK4T5, Piece ID: GVIETIACIWYR6PJL6IHA6CP226CJL37YJ2OET4ZECRU36MMI4I7Q) context deadline exceeded
======LOGS======
Dec 28 03:53:04.073632789 (540h24m3.944715834s) [upload=473925]: Starting upload
Dec 28 03:53:04.073724814 (540h24m3.94480786s) [upload=473925]: Done starting upload
Dec 28 03:53:04.073828211 (540h24m3.944911256s) [upload=473925]: Uploading segments...
Dec 28 03:53:04.073962767 (540h24m3.945045812s) [upload=473925]: Got next segment. Inline: false
Dec 28 03:53:04.073999807 (540h24m3.945082842s) [upload=473925]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 28 03:53:04.074004667 (540h24m3.945087702s) [upload=473925]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 28 03:53:04.318362666 (540h24m4.189445722s) [seg_pos={0 0}, upload=473925]: Begin upload segment...
Dec 28 03:53:04.318425976 (540h24m4.189509021s) [seg_pos={0 0}, upload=473925]: Done begin upload segment.
Dec 28 03:53:04.318464541 (540h24m4.189547586s) [noise=true , address=77.90.31.14:4664 , piece_id=EBANH3SP7FG5YNVQCTALZFSI3JZMEK3UKEGQEWAEG4PZVCFHOIQQ, seg_pos={0 0}, upload=473925]: Uploading piece...
Dec 28 03:53:04.319587318 (540h24m4.190670363s) [noise=true , address=145.79.227.4:4187 , piece_id=2QNAPJOYG5CJWMC4DWJ266ZKRKF32TQLX4CFYNOH4BDXOEGWVXIA, seg_pos={0 0}, upload=473925]: Uploading piece...
Dec 28 03:53:04.320754610 (540h24m4.191837656s) [noise=true , address=145.79.227.51:4410 , piece_id=B4KRJWKFECY4M7GRWCBVCW3EHMFCPCJAQJZBYFTDYAQKERI3T73A, seg_pos={0 0}, upload=473925]: Uploading piece...
Dec 28 03:53:04.322061279 (540h24m4.193144324s) [noise=true , address=145.79.227.45:4231 , piece_id=G44P7NGG7AHY6QBBBDMKBJUODO6T2PETJDKJV66P5RX3HUI5EY5Q, seg_pos={0 0}, upload=473925]: Uploading piece...
Dec 28 03:53:04.323593146 (540h24m4.194676191s) [noise=true , address=145.79.227.5:4223 , piece_id=ZAXJG4JWQRXWWET2FGBJUCE3BP4C4EANJV73WUG7KDCAEKQQUELQ, seg_pos={0 0}, upload=473925]: Uploading piece...
Dec 28 03:53:04.325065940 (540h24m4.196148985s) [noise=true , address=145.79.227.30:4761 , piece_id=6VVUECBRZV3XHBKAPONOSFLZH7MB4RCFP35RJG3W2AB5N7XUGV4Q, seg_pos={0 0}, upload=473925]: Uploading piece...
Dec 28 03:53:04.325561013 (540h24m4.196644048s) [noise=true , address=145.79.227.11:4729 , piece_id=IYLXC2VWSSHSSIDGOGLEKS5EHUDTCAQZZ3J6NL2ZGBAO6IXXSQUA, seg_pos={0 0}, upload=473925]: Uploading piece...
Dec 28 03:53:04.327363034 (540h24m4.198446079s) [noise=true , address=145.79.227.19:4078 , piece_id=ZYLONIRIM3JDMKRT3UMD5AAWGKPJHPCDYI5MH3ESZO54SFWRCC5A, seg_pos={0 0}, upload=473925]: Uploading piece...
Dec 28 03:53:04.327971924 (540h24m4.199054959s) [noise=true , address=45.91.116.6:4976 , piece_id=U7BVCY6UKFJFLCRF3DXLPPF2MUCF7G465VAGPNG35AEMYQOS26GA, seg_pos={0 0}, upload=473925]: Uploading piece...
Dec 28 03:53:04.328691724 (540h24m4.199774759s) [noise=true , address=145.79.227.18:4007 , piece_id=Z662WKMGYAV2UB2LGAUJIA4YLJIZ5IRL2U57F3H7TCNOF62PURHA, seg_pos={0 0}, upload=473925]: Uploading piece...
Dec 28 03:53:04.329385465 (540h24m4.20046851s) [noise=true , address=77.90.31.17:4773 , piece_id=XKJ3IAQ4QGNWYFVAGVPIK6O7KB5KRTR2LVS2X2UH4OHANDVLLL3Q, seg_pos={0 0}, upload=473925]: Uploading piece...
Dec 28 03:53:04.330004835 (540h24m4.20108788s) [noise=true , address=145.79.227.50:4375 , piece_id=GVIETIACIWYR6PJL6IHA6CP226CJL37YJ2OET4ZECRU36MMI4I7Q, seg_pos={0 0}, upload=473925]: Uploading piece...
Dec 28 03:53:04.330185779 (540h24m4.201268824s) [noise=true , address=45.91.116.13:4153 , piece_id=7WYVTCLBCV5JOJZ523DUCBWYGZZII2VGUL2IUCAJF32QI4TGTNOA, seg_pos={0 0}, upload=473925]: Uploading piece...
Dec 28 03:53:04.330934195 (540h24m4.20201723s) [noise=true , address=145.79.227.46:4601 , piece_id=YXL24FDMISTWR2ZPSLYZ5BJOBKBNNQYNWVMOUI6HKQL6GVBSWVDQ, seg_pos={0 0}, upload=473925]: Uploading piece...
Dec 28 03:53:04.331420952 (540h24m4.202503997s) [noise=true , address=45.91.116.8:4048 , piece_id=XSDJIS4KA72F3676LHIQKKXU3AQB5TER3ILEKOQVIGSCELH6CE6A, seg_pos={0 0}, upload=473925]: Uploading piece...
Dec 28 03:53:04.331887540 (540h24m4.202970585s) [noise=true , address=45.91.116.12:4227 , piece_id=2TI2M3K7DAQSTO7GGUYN4SL6D6RPTUWZ7QRNHEHKNZRVZ2TGZZRQ, seg_pos={0 0}, upload=473925]: Uploading piece...
Dec 28 03:53:06.035309407 (540h24m5.906392442s) [upload=473925]: Next returned nil segment
Dec 28 03:53:06.035318415 (540h24m5.90640145s) [upload=473925]: Waiting for error group managing segments...
Dec 28 03:53:06.334442147 (540h24m6.205525193s) [noise=true , address=145.79.227.19:4078 , piece_id=ZYLONIRIM3JDMKRT3UMD5AAWGKPJHPCDYI5MH3ESZO54SFWRCC5A, seg_pos={0 0}, upload=473925]: Done uploading piece. err: <nil>
Dec 28 03:53:06.348259298 (540h24m6.219342333s) [noise=true , address=145.79.227.46:4601 , piece_id=YXL24FDMISTWR2ZPSLYZ5BJOBKBNNQYNWVMOUI6HKQL6GVBSWVDQ, seg_pos={0 0}, upload=473925]: Done uploading piece. err: <nil>
Dec 28 03:53:06.355542426 (540h24m6.226625461s) [noise=true , address=145.79.227.5:4223 , piece_id=ZAXJG4JWQRXWWET2FGBJUCE3BP4C4EANJV73WUG7KDCAEKQQUELQ, seg_pos={0 0}, upload=473925]: Done uploading piece. err: <nil>
Dec 28 03:53:06.358523523 (540h24m6.229606568s) [noise=true , address=145.79.227.4:4187 , piece_id=2QNAPJOYG5CJWMC4DWJ266ZKRKF32TQLX4CFYNOH4BDXOEGWVXIA, seg_pos={0 0}, upload=473925]: Done uploading piece. err: <nil>
Dec 28 03:53:06.358731308 (540h24m6.229814354s) [noise=true , address=145.79.227.51:4410 , piece_id=B4KRJWKFECY4M7GRWCBVCW3EHMFCPCJAQJZBYFTDYAQKERI3T73A, seg_pos={0 0}, upload=473925]: Done uploading piece. err: <nil>
Dec 28 03:53:06.364868775 (540h24m6.23595181s) [noise=true , address=145.79.227.50:4375 , piece_id=GVIETIACIWYR6PJL6IHA6CP226CJL37YJ2OET4ZECRU36MMI4I7Q, seg_pos={0 0}, upload=473925]: Done uploading piece. err: <nil>
Dec 28 03:53:06.371362481 (540h24m6.242445516s) [noise=true , address=145.79.227.11:4729 , piece_id=IYLXC2VWSSHSSIDGOGLEKS5EHUDTCAQZZ3J6NL2ZGBAO6IXXSQUA, seg_pos={0 0}, upload=473925]: Done uploading piece. err: <nil>
Dec 28 03:53:06.636977377 (540h24m6.508060422s) [noise=true , address=77.90.31.14:4664 , piece_id=EBANH3SP7FG5YNVQCTALZFSI3JZMEK3UKEGQEWAEG4PZVCFHOIQQ, seg_pos={0 0}, upload=473925]: Done uploading piece. err: <nil>
Dec 28 03:53:06.672895521 (540h24m6.543978566s) [noise=true , address=45.91.116.13:4153 , piece_id=7WYVTCLBCV5JOJZ523DUCBWYGZZII2VGUL2IUCAJF32QI4TGTNOA, seg_pos={0 0}, upload=473925]: Done uploading piece. err: <nil>
Dec 28 03:53:06.674418421 (540h24m6.545501456s) [noise=true , address=45.91.116.8:4048 , piece_id=XSDJIS4KA72F3676LHIQKKXU3AQB5TER3ILEKOQVIGSCELH6CE6A, seg_pos={0 0}, upload=473925]: Done uploading piece. err: <nil>
Dec 28 03:53:06.899082400 (540h24m6.770165445s) [noise=true , address=45.91.116.12:4227 , piece_id=2TI2M3K7DAQSTO7GGUYN4SL6D6RPTUWZ7QRNHEHKNZRVZ2TGZZRQ, seg_pos={0 0}, upload=473925]: Done uploading piece. err: <nil>
Dec 28 03:53:06.952737939 (540h24m6.823820984s) [noise=true , address=145.79.227.18:4007 , piece_id=Z662WKMGYAV2UB2LGAUJIA4YLJIZ5IRL2U57F3H7TCNOF62PURHA, seg_pos={0 0}, upload=473925]: Done uploading piece. err: <nil>
Dec 28 03:53:07.067362283 (540h24m6.938445319s) [noise=true , address=145.79.227.45:4231 , piece_id=G44P7NGG7AHY6QBBBDMKBJUODO6T2PETJDKJV66P5RX3HUI5EY5Q, seg_pos={0 0}, upload=473925]: Done uploading piece. err: <nil>
Dec 28 03:53:07.067630724 (540h24m6.938713759s) [seg_pos={0 0}, upload=473925]: Segment reached optimal threshold of 13 pieces.
Dec 28 03:53:07.067965562 (540h24m6.939048597s) [noise=true , address=145.79.227.30:4761 , piece_id=6VVUECBRZV3XHBKAPONOSFLZH7MB4RCFP35RJG3W2AB5N7XUGV4Q, seg_pos={0 0}, upload=473925]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF
Dec 28 03:53:07.067975741 (540h24m6.939058776s) [noise=true , address=145.79.227.30:4761 , piece_id=6VVUECBRZV3XHBKAPONOSFLZH7MB4RCFP35RJG3W2AB5N7XUGV4Q, seg_pos={0 0}, upload=473925]: Long tail cancellation due to optimal threshold
Dec 28 03:53:07.068203986 (540h24m6.939287021s) [noise=true , address=45.91.116.6:4976 , piece_id=U7BVCY6UKFJFLCRF3DXLPPF2MUCF7G465VAGPNG35AEMYQOS26GA, seg_pos={0 0}, upload=473925]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: write tcp 188.166.225.170:49568->45.91.116.6:4976: use of closed network connection; write tcp 188.166.225.170:49568->45.91.116.6:4976: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49568->45.91.116.6:4976: use of closed network connection
Dec 28 03:53:07.068210578 (540h24m6.939293613s) [noise=true , address=45.91.116.6:4976 , piece_id=U7BVCY6UKFJFLCRF3DXLPPF2MUCF7G465VAGPNG35AEMYQOS26GA, seg_pos={0 0}, upload=473925]: Long tail cancellation due to optimal threshold
Dec 28 03:53:07.068490010 (540h24m6.939573056s) [noise=true , address=77.90.31.17:4773 , piece_id=XKJ3IAQ4QGNWYFVAGVPIK6O7KB5KRTR2LVS2X2UH4OHANDVLLL3Q, seg_pos={0 0}, upload=473925]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ): protocol: expected piece hash; context canceled; EOF
Dec 28 03:53:07.068502103 (540h24m6.939585168s) [noise=true , address=77.90.31.17:4773 , piece_id=XKJ3IAQ4QGNWYFVAGVPIK6O7KB5KRTR2LVS2X2UH4OHANDVLLL3Q, seg_pos={0 0}, upload=473925]: Long tail cancellation due to optimal threshold
Dec 28 03:53:07.068548693 (540h24m6.939631728s) [seg_pos={0 0}, upload=473925]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 28 03:53:07.068632872 (540h24m6.939715907s) [upload=473925]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 03:53:07.068638884 (540h24m6.939721919s) [upload=473925]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 03:53:07.360205988 (540h24m7.231289023s) [upload=473925]: Done uploading segments...
Dec 28 03:53:07.360226496 (540h24m7.231309542s) [upload=473925]: Upload finished. err: <nil>
Duration: 30.003623565s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [...................X...X....................................] duration:
28.646609596s trace:
https://jaeger.dev.storj.io/trace/54422f75ff432c0
FAIL (2025-12-28 03:53:27 UTC)
orbiter warning:
operation: DeleteObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
, piece_id=SPQSUVHIHI32ZMIRE4NHSNJCMWS4NS6ZZPYF6FV7FQH6VADQIV5A, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:35.720215866 (539h54m35.591298911s) [noise=true , address=145.79.227.43:4187 , piece_id=MRILXYFSWVSH35ZCCFZWLEXENWJII6G6LOR2N277GIBMI4PXPPTQ, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:35.720567665 (539h54m35.5916507s) [noise=true , address=145.79.227.33:4896 , piece_id=7NRM7I4IDHW2WGOCG3B6L6HN74ANXWDJCFRPBM4BCHWJELSNNUXA, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:35.720883607 (539h54m35.591966652s) [noise=true , address=145.79.227.40:4062 , piece_id=RCVKQLDSWIWX2FQBTUFPEUXQZLP2C3ZCLV2JWHOV7CMJ7ZBNWZNQ, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:35.721874904 (539h54m35.592957949s) [noise=true , address=64.49.13.140:4513 , piece_id=Y3HW5I2FX2CWSPVUS7CU3JPNN2RHP2ASL757AUYINBKRKCLXAJGQ, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:35.722249658 (539h54m35.593332693s) [noise=true , address=64.49.13.169:4715 , piece_id=FRVNWFP2AE4DXE4RKZRE6WTK43T74BTTYM2U7E4P3B4XZIMC2QMQ, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:35.722786881 (539h54m35.593869926s) [noise=true , address=77.90.31.14:4673 , piece_id=ZIB5VOFL7PJSRFDV3HP5XAKYHOJXRVHCYDPIHKRYPNM5ROAY5DTQ, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:35.723112100 (539h54m35.594195135s) [noise=true , address=77.90.31.17:4779 , piece_id=TLU5ZLZRVAMGQLSE2LLNIUE4WOWJBMFJGFQKC3KOUJ23V3UVFMRA, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:35.723440756 (539h54m35.594523791s) [noise=true , address=145.79.227.37:4569 , piece_id=DUXSQUT36XUF66ZVQ2V7ZWNXA6JZKBDO5RXT3QVT26GE2FTH3QXQ, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:35.723849544 (539h54m35.594932589s) [noise=true , address=145.79.227.27:4723 , piece_id=4DECSGDNZLXNQVPU7QALLPFQYDO7MOG6TIBNWQ5UHNVCIHB4GUYQ, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:35.724241290 (539h54m35.595324335s) [noise=true , address=77.90.31.11:4587 , piece_id=6467VZCLWHJTZ36ENMEOEOU2E4MAUCDL2XHNAOMJTCCTUSN33NFA, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:35.724607808 (539h54m35.595690854s) [noise=true , address=145.79.227.47:4272 , piece_id=FL2U4W2YCV5ANDVFHB7H4TFT3SYTBGNLI3KZDI7MY2K6NTSMRUAA, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:35.724974035 (539h54m35.59605707s) [noise=true , address=145.79.227.18:4040 , piece_id=W55GK3QDHRHNR6YJ7JKVZMIEKFQLZSTXKSQ5OMW3U7XRI44EM6RA, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:35.725302281 (539h54m35.596385316s) [noise=true , address=145.79.227.38:4989 , piece_id=OUINZ6BXOLFCDNCXMX7PPFSKWMJ7IPZWO7DAGLGVV2AB7UHO3JYQ, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:35.726824420 (539h54m35.597907455s) [noise=true , address=145.79.227.26:4669 , piece_id=5D7SGVO274FEZ4JVESTMUYQ3RZ3DJA5DL2QZV5YKMXEB4TM6PHAA, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:37.063163473 (539h54m36.934246518s) [upload=473490]: Next returned nil segment
Dec 28 03:23:37.063176499 (539h54m36.934259534s) [upload=473490]: Waiting for error group managing segments...
Dec 28 03:23:37.416516311 (539h54m37.287599376s) [noise=true , address=64.49.13.140:4513 , piece_id=Y3HW5I2FX2CWSPVUS7CU3JPNN2RHP2ASL757AUYINBKRKCLXAJGQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil>
Dec 28 03:23:37.606629586 (539h54m37.477712622s) [noise=true , address=145.79.227.27:4723 , piece_id=4DECSGDNZLXNQVPU7QALLPFQYDO7MOG6TIBNWQ5UHNVCIHB4GUYQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil>
Dec 28 03:23:37.617730204 (539h54m37.488813249s) [noise=true , address=145.79.227.26:4669 , piece_id=5D7SGVO274FEZ4JVESTMUYQ3RZ3DJA5DL2QZV5YKMXEB4TM6PHAA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil>
Dec 28 03:23:37.632542950 (539h54m37.503625995s) [noise=true , address=145.79.227.38:4989 , piece_id=OUINZ6BXOLFCDNCXMX7PPFSKWMJ7IPZWO7DAGLGVV2AB7UHO3JYQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil>
Dec 28 03:23:37.643987531 (539h54m37.515070566s) [noise=true , address=145.79.227.47:4272 , piece_id=FL2U4W2YCV5ANDVFHB7H4TFT3SYTBGNLI3KZDI7MY2K6NTSMRUAA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil>
Dec 28 03:23:37.652913619 (539h54m37.523996654s) [noise=true , address=145.79.227.18:4040 , piece_id=W55GK3QDHRHNR6YJ7JKVZMIEKFQLZSTXKSQ5OMW3U7XRI44EM6RA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil>
Dec 28 03:23:37.661966357 (539h54m37.533049392s) [noise=true , address=145.79.227.37:4569 , piece_id=DUXSQUT36XUF66ZVQ2V7ZWNXA6JZKBDO5RXT3QVT26GE2FTH3QXQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil>
Dec 28 03:23:37.778627848 (539h54m37.649710893s) [noise=true , address=77.90.31.11:4561 , piece_id=HIUCZRRZOMU4RBYZU5SADJAOS4RVML2I6PILIX2V65RS22EBN42Q, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil>
Dec 28 03:23:37.815974812 (539h54m37.687057857s) [noise=true , address=45.91.116.14:4217 , piece_id=SPQSUVHIHI32ZMIRE4NHSNJCMWS4NS6ZZPYF6FV7FQH6VADQIV5A, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil>
Dec 28 03:23:37.861731780 (539h54m37.732814815s) [noise=true , address=77.90.31.14:4673 , piece_id=ZIB5VOFL7PJSRFDV3HP5XAKYHOJXRVHCYDPIHKRYPNM5ROAY5DTQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil>
Dec 28 03:23:38.093273211 (539h54m37.964356256s) [noise=true , address=77.90.31.11:4587 , piece_id=6467VZCLWHJTZ36ENMEOEOU2E4MAUCDL2XHNAOMJTCCTUSN33NFA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil>
Dec 28 03:23:38.267694395 (539h54m38.13877745s) [noise=true , address=77.90.31.17:4779 , piece_id=TLU5ZLZRVAMGQLSE2LLNIUE4WOWJBMFJGFQKC3KOUJ23V3UVFMRA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil>
Dec 28 03:23:38.540947761 (539h54m38.412030796s) [noise=true , address=145.79.227.40:4062 , piece_id=RCVKQLDSWIWX2FQBTUFPEUXQZLP2C3ZCLV2JWHOV7CMJ7ZBNWZNQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil>
Dec 28 03:23:38.815393527 (539h54m38.686476562s) [noise=true , address=145.79.227.43:4187 , piece_id=MRILXYFSWVSH35ZCCFZWLEXENWJII6G6LOR2N277GIBMI4PXPPTQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil>
Dec 28 03:23:38.815412894 (539h54m38.686495939s) [seg_pos={0 0}, upload=473490]: Segment reached optimal threshold of 14 pieces.
Dec 28 03:23:38.815594951 (539h54m38.686677986s) [noise=true , address=64.49.13.169:4715 , piece_id=FRVNWFP2AE4DXE4RKZRE6WTK43T74BTTYM2U7E4P3B4XZIMC2QMQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:43502->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:43502->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43502->64.49.13.169:4715: use of closed network connection
Dec 28 03:23:38.815601032 (539h54m38.686684077s) [noise=true , address=64.49.13.169:4715 , piece_id=FRVNWFP2AE4DXE4RKZRE6WTK43T74BTTYM2U7E4P3B4XZIMC2QMQ, seg_pos={0 0}, upload=473490]: Long tail cancellation due to optimal threshold
Dec 28 03:23:38.815826983 (539h54m38.686910018s) [noise=true , address=145.79.227.41:4118 , piece_id=C7JJ3NHR5355M64G2EF2WWERQVGWR243RQHDV3GBAIC3A66JDMHQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128w37UHxJB9R49iwmHbATt3bXUb1FMTQJ3Wjka7So7eqKWg4iF): protocol: expected piece hash; context canceled; EOF
Dec 28 03:23:38.815834507 (539h54m38.686917542s) [noise=true , address=145.79.227.41:4118 , piece_id=C7JJ3NHR5355M64G2EF2WWERQVGWR243RQHDV3GBAIC3A66JDMHQ, seg_pos={0 0}, upload=473490]: Long tail cancellation due to optimal threshold
Dec 28 03:23:38.815890433 (539h54m38.686973468s) [noise=true , address=145.79.227.33:4896 , piece_id=7NRM7I4IDHW2WGOCG3B6L6HN74ANXWDJCFRPBM4BCHWJELSNNUXA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ndSgjgfZMVQni5tw1ugdTrhJt4c4NagdvgYTbSbjatYYtfu1o): protocol: expected piece hash; context canceled; EOF
Dec 28 03:23:38.815914619 (539h54m38.686997654s) [noise=true , address=145.79.227.33:4896 , piece_id=7NRM7I4IDHW2WGOCG3B6L6HN74ANXWDJCFRPBM4BCHWJELSNNUXA, seg_pos={0 0}, upload=473490]: Long tail cancellation due to optimal threshold
Dec 28 03:23:38.816105613 (539h54m38.687188648s) [noise=true , address=145.79.227.28:4731 , piece_id=EPW3LUA3EWRCURHZ66NCRCA36CNIW4GYQMFNKPZZTRCB3YBGTKJA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NQzTwVYQa6ZzxAHQm1apkwWnqJstTnAJQyHKhBzC2jTN9kEVK): protocol: write tcp 188.166.225.170:58940->145.79.227.28:4731: use of closed network connection; write tcp 188.166.225.170:58940->145.79.227.28:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58940->145.79.227.28:4731: use of closed network connection
Dec 28 03:23:38.816112426 (539h54m38.687195461s) [noise=true , address=145.79.227.28:4731 , piece_id=EPW3LUA3EWRCURHZ66NCRCA36CNIW4GYQMFNKPZZTRCB3YBGTKJA, seg_pos={0 0}, upload=473490]: Long tail cancellation due to optimal threshold
Dec 28 03:23:38.816128436 (539h54m38.687211481s) [seg_pos={0 0}, upload=473490]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 28 03:23:38.816257582 (539h54m38.687340627s) [upload=473490]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 03:23:38.816265778 (539h54m38.687348813s) [upload=473490]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 03:23:39.111115407 (539h54m38.982198442s) [upload=473490]: Done uploading segments...
Dec 28 03:23:39.111138972 (539h54m38.982222008s) [upload=473490]: Upload finished. err: <nil>
Duration: 28.646627841s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [....................X...X...................................] duration:
28.000148665s trace:
https://jaeger.dev.storj.io/trace/54422f75ff432c0
FAIL (2025-12-28 03:52:58 UTC)
orbiter warning:
operation: DeleteObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
, piece_id=SPQSUVHIHI32ZMIRE4NHSNJCMWS4NS6ZZPYF6FV7FQH6VADQIV5A, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:35.720215866 (539h54m35.591298911s) [noise=true , address=145.79.227.43:4187 , piece_id=MRILXYFSWVSH35ZCCFZWLEXENWJII6G6LOR2N277GIBMI4PXPPTQ, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:35.720567665 (539h54m35.5916507s) [noise=true , address=145.79.227.33:4896 , piece_id=7NRM7I4IDHW2WGOCG3B6L6HN74ANXWDJCFRPBM4BCHWJELSNNUXA, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:35.720883607 (539h54m35.591966652s) [noise=true , address=145.79.227.40:4062 , piece_id=RCVKQLDSWIWX2FQBTUFPEUXQZLP2C3ZCLV2JWHOV7CMJ7ZBNWZNQ, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:35.721874904 (539h54m35.592957949s) [noise=true , address=64.49.13.140:4513 , piece_id=Y3HW5I2FX2CWSPVUS7CU3JPNN2RHP2ASL757AUYINBKRKCLXAJGQ, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:35.722249658 (539h54m35.593332693s) [noise=true , address=64.49.13.169:4715 , piece_id=FRVNWFP2AE4DXE4RKZRE6WTK43T74BTTYM2U7E4P3B4XZIMC2QMQ, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:35.722786881 (539h54m35.593869926s) [noise=true , address=77.90.31.14:4673 , piece_id=ZIB5VOFL7PJSRFDV3HP5XAKYHOJXRVHCYDPIHKRYPNM5ROAY5DTQ, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:35.723112100 (539h54m35.594195135s) [noise=true , address=77.90.31.17:4779 , piece_id=TLU5ZLZRVAMGQLSE2LLNIUE4WOWJBMFJGFQKC3KOUJ23V3UVFMRA, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:35.723440756 (539h54m35.594523791s) [noise=true , address=145.79.227.37:4569 , piece_id=DUXSQUT36XUF66ZVQ2V7ZWNXA6JZKBDO5RXT3QVT26GE2FTH3QXQ, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:35.723849544 (539h54m35.594932589s) [noise=true , address=145.79.227.27:4723 , piece_id=4DECSGDNZLXNQVPU7QALLPFQYDO7MOG6TIBNWQ5UHNVCIHB4GUYQ, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:35.724241290 (539h54m35.595324335s) [noise=true , address=77.90.31.11:4587 , piece_id=6467VZCLWHJTZ36ENMEOEOU2E4MAUCDL2XHNAOMJTCCTUSN33NFA, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:35.724607808 (539h54m35.595690854s) [noise=true , address=145.79.227.47:4272 , piece_id=FL2U4W2YCV5ANDVFHB7H4TFT3SYTBGNLI3KZDI7MY2K6NTSMRUAA, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:35.724974035 (539h54m35.59605707s) [noise=true , address=145.79.227.18:4040 , piece_id=W55GK3QDHRHNR6YJ7JKVZMIEKFQLZSTXKSQ5OMW3U7XRI44EM6RA, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:35.725302281 (539h54m35.596385316s) [noise=true , address=145.79.227.38:4989 , piece_id=OUINZ6BXOLFCDNCXMX7PPFSKWMJ7IPZWO7DAGLGVV2AB7UHO3JYQ, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:35.726824420 (539h54m35.597907455s) [noise=true , address=145.79.227.26:4669 , piece_id=5D7SGVO274FEZ4JVESTMUYQ3RZ3DJA5DL2QZV5YKMXEB4TM6PHAA, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:37.063163473 (539h54m36.934246518s) [upload=473490]: Next returned nil segment
Dec 28 03:23:37.063176499 (539h54m36.934259534s) [upload=473490]: Waiting for error group managing segments...
Dec 28 03:23:37.416516311 (539h54m37.287599376s) [noise=true , address=64.49.13.140:4513 , piece_id=Y3HW5I2FX2CWSPVUS7CU3JPNN2RHP2ASL757AUYINBKRKCLXAJGQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil>
Dec 28 03:23:37.606629586 (539h54m37.477712622s) [noise=true , address=145.79.227.27:4723 , piece_id=4DECSGDNZLXNQVPU7QALLPFQYDO7MOG6TIBNWQ5UHNVCIHB4GUYQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil>
Dec 28 03:23:37.617730204 (539h54m37.488813249s) [noise=true , address=145.79.227.26:4669 , piece_id=5D7SGVO274FEZ4JVESTMUYQ3RZ3DJA5DL2QZV5YKMXEB4TM6PHAA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil>
Dec 28 03:23:37.632542950 (539h54m37.503625995s) [noise=true , address=145.79.227.38:4989 , piece_id=OUINZ6BXOLFCDNCXMX7PPFSKWMJ7IPZWO7DAGLGVV2AB7UHO3JYQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil>
Dec 28 03:23:37.643987531 (539h54m37.515070566s) [noise=true , address=145.79.227.47:4272 , piece_id=FL2U4W2YCV5ANDVFHB7H4TFT3SYTBGNLI3KZDI7MY2K6NTSMRUAA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil>
Dec 28 03:23:37.652913619 (539h54m37.523996654s) [noise=true , address=145.79.227.18:4040 , piece_id=W55GK3QDHRHNR6YJ7JKVZMIEKFQLZSTXKSQ5OMW3U7XRI44EM6RA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil>
Dec 28 03:23:37.661966357 (539h54m37.533049392s) [noise=true , address=145.79.227.37:4569 , piece_id=DUXSQUT36XUF66ZVQ2V7ZWNXA6JZKBDO5RXT3QVT26GE2FTH3QXQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil>
Dec 28 03:23:37.778627848 (539h54m37.649710893s) [noise=true , address=77.90.31.11:4561 , piece_id=HIUCZRRZOMU4RBYZU5SADJAOS4RVML2I6PILIX2V65RS22EBN42Q, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil>
Dec 28 03:23:37.815974812 (539h54m37.687057857s) [noise=true , address=45.91.116.14:4217 , piece_id=SPQSUVHIHI32ZMIRE4NHSNJCMWS4NS6ZZPYF6FV7FQH6VADQIV5A, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil>
Dec 28 03:23:37.861731780 (539h54m37.732814815s) [noise=true , address=77.90.31.14:4673 , piece_id=ZIB5VOFL7PJSRFDV3HP5XAKYHOJXRVHCYDPIHKRYPNM5ROAY5DTQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil>
Dec 28 03:23:38.093273211 (539h54m37.964356256s) [noise=true , address=77.90.31.11:4587 , piece_id=6467VZCLWHJTZ36ENMEOEOU2E4MAUCDL2XHNAOMJTCCTUSN33NFA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil>
Dec 28 03:23:38.267694395 (539h54m38.13877745s) [noise=true , address=77.90.31.17:4779 , piece_id=TLU5ZLZRVAMGQLSE2LLNIUE4WOWJBMFJGFQKC3KOUJ23V3UVFMRA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil>
Dec 28 03:23:38.540947761 (539h54m38.412030796s) [noise=true , address=145.79.227.40:4062 , piece_id=RCVKQLDSWIWX2FQBTUFPEUXQZLP2C3ZCLV2JWHOV7CMJ7ZBNWZNQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil>
Dec 28 03:23:38.815393527 (539h54m38.686476562s) [noise=true , address=145.79.227.43:4187 , piece_id=MRILXYFSWVSH35ZCCFZWLEXENWJII6G6LOR2N277GIBMI4PXPPTQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil>
Dec 28 03:23:38.815412894 (539h54m38.686495939s) [seg_pos={0 0}, upload=473490]: Segment reached optimal threshold of 14 pieces.
Dec 28 03:23:38.815594951 (539h54m38.686677986s) [noise=true , address=64.49.13.169:4715 , piece_id=FRVNWFP2AE4DXE4RKZRE6WTK43T74BTTYM2U7E4P3B4XZIMC2QMQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:43502->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:43502->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43502->64.49.13.169:4715: use of closed network connection
Dec 28 03:23:38.815601032 (539h54m38.686684077s) [noise=true , address=64.49.13.169:4715 , piece_id=FRVNWFP2AE4DXE4RKZRE6WTK43T74BTTYM2U7E4P3B4XZIMC2QMQ, seg_pos={0 0}, upload=473490]: Long tail cancellation due to optimal threshold
Dec 28 03:23:38.815826983 (539h54m38.686910018s) [noise=true , address=145.79.227.41:4118 , piece_id=C7JJ3NHR5355M64G2EF2WWERQVGWR243RQHDV3GBAIC3A66JDMHQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128w37UHxJB9R49iwmHbATt3bXUb1FMTQJ3Wjka7So7eqKWg4iF): protocol: expected piece hash; context canceled; EOF
Dec 28 03:23:38.815834507 (539h54m38.686917542s) [noise=true , address=145.79.227.41:4118 , piece_id=C7JJ3NHR5355M64G2EF2WWERQVGWR243RQHDV3GBAIC3A66JDMHQ, seg_pos={0 0}, upload=473490]: Long tail cancellation due to optimal threshold
Dec 28 03:23:38.815890433 (539h54m38.686973468s) [noise=true , address=145.79.227.33:4896 , piece_id=7NRM7I4IDHW2WGOCG3B6L6HN74ANXWDJCFRPBM4BCHWJELSNNUXA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ndSgjgfZMVQni5tw1ugdTrhJt4c4NagdvgYTbSbjatYYtfu1o): protocol: expected piece hash; context canceled; EOF
Dec 28 03:23:38.815914619 (539h54m38.686997654s) [noise=true , address=145.79.227.33:4896 , piece_id=7NRM7I4IDHW2WGOCG3B6L6HN74ANXWDJCFRPBM4BCHWJELSNNUXA, seg_pos={0 0}, upload=473490]: Long tail cancellation due to optimal threshold
Dec 28 03:23:38.816105613 (539h54m38.687188648s) [noise=true , address=145.79.227.28:4731 , piece_id=EPW3LUA3EWRCURHZ66NCRCA36CNIW4GYQMFNKPZZTRCB3YBGTKJA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NQzTwVYQa6ZzxAHQm1apkwWnqJstTnAJQyHKhBzC2jTN9kEVK): protocol: write tcp 188.166.225.170:58940->145.79.227.28:4731: use of closed network connection; write tcp 188.166.225.170:58940->145.79.227.28:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58940->145.79.227.28:4731: use of closed network connection
Dec 28 03:23:38.816112426 (539h54m38.687195461s) [noise=true , address=145.79.227.28:4731 , piece_id=EPW3LUA3EWRCURHZ66NCRCA36CNIW4GYQMFNKPZZTRCB3YBGTKJA, seg_pos={0 0}, upload=473490]: Long tail cancellation due to optimal threshold
Dec 28 03:23:38.816128436 (539h54m38.687211481s) [seg_pos={0 0}, upload=473490]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 28 03:23:38.816257582 (539h54m38.687340627s) [upload=473490]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 03:23:38.816265778 (539h54m38.687348813s) [upload=473490]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 03:23:39.111115407 (539h54m38.982198442s) [upload=473490]: Done uploading segments...
Dec 28 03:23:39.111138972 (539h54m38.982222008s) [upload=473490]: Upload finished. err: <nil>
Duration: 28.0001938s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
21 out of 60 is failed (threshold=6) [...X.....XX.....X..X.X....X.XXXX....X.X.X...X..X.X.X..X...XX] duration:
5.001855991s trace:
https://jaeger.dev.storj.io/trace/7e7d6d3ddfeaade0
FAIL (2025-12-28 03:52:42 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001880377s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
51 out of 60 is failed (threshold=3) [.XX..XXXXXX.XXXXXXXXXXXXXXXXX.XXXXX.XXXXXXXXXXXXXXXXX..XXXX.] duration:
6.112778277s trace:
https://jaeger.dev.storj.io/trace/1f754c262e8a6f59
FAIL (2025-12-28 03:52:38 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA, Piece ID: 3SFQIZV6KAIGUF4DEPCJ2OKEAOXHDE6E7MGB4BOYRXL6OSSNJ5OQ) context deadline exceeded; piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: QW35GCUQ5KJBLUKR7ZUW3PUK44YIJVBMYAHIXSGUGDPCHJQCCZUA) context deadline exceeded
======LOGS======
Dec 28 03:51:59.409570400 (540h22m59.280653445s) [upload=473908]: Starting upload
Dec 28 03:51:59.409624112 (540h22m59.280707157s) [upload=473908]: Done starting upload
Dec 28 03:51:59.410221440 (540h22m59.281304475s) [upload=473908]: Uploading segments...
Dec 28 03:51:59.410295541 (540h22m59.281378576s) [upload=473908]: Got next segment. Inline: false
Dec 28 03:51:59.410427493 (540h22m59.281510538s) [upload=473908]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 28 03:51:59.410432622 (540h22m59.281515667s) [upload=473908]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 28 03:51:59.654512031 (540h22m59.525595066s) [seg_pos={0 0}, upload=473908]: Begin upload segment...
Dec 28 03:51:59.654571474 (540h22m59.525654509s) [seg_pos={0 0}, upload=473908]: Done begin upload segment.
Dec 28 03:51:59.654674801 (540h22m59.525757846s) [noise=true , address=45.91.116.7:4011 , piece_id=QW35GCUQ5KJBLUKR7ZUW3PUK44YIJVBMYAHIXSGUGDPCHJQCCZUA, seg_pos={0 0}, upload=473908]: Uploading piece...
Dec 28 03:51:59.654679921 (540h22m59.525763006s) [noise=true , address=45.91.116.13:4156 , piece_id=P4IVKHNZDQNNRBTLE4AO6DRO5NEGVHW6K3AYJ37FTEICYVVB5E4A, seg_pos={0 0}, upload=473908]: Uploading piece...
Dec 28 03:51:59.655256279 (540h22m59.526339314s) [noise=true , address=145.79.227.2:4115 , piece_id=ZDZ4LWRF3C2PWGUG67CN2URYX3YCE6KUP5MUGO3335AFEWQJK3WA, seg_pos={0 0}, upload=473908]: Uploading piece...
Dec 28 03:51:59.655482629 (540h22m59.526565664s) [noise=true , address=77.90.31.19:4842 , piece_id=YH224UFOU6AFIDGZNEAYG4JQYOWGWPX3PIQJU25GOEC6UB5S4XNQ, seg_pos={0 0}, upload=473908]: Uploading piece...
Dec 28 03:51:59.655614020 (540h22m59.526697065s) [noise=true , address=45.91.116.5:4300 , piece_id=LNYOIQ3J6MI4Z4FUAZJTLSI7LFC3U5ZF3YHAASMP2TH4TO3ANDSQ, seg_pos={0 0}, upload=473908]: Uploading piece...
Dec 28 03:51:59.655776630 (540h22m59.526859675s) [noise=true , address=145.79.227.39:4014 , piece_id=56263GZBKICQ652BAVE6YO4RY23W3L6BBVR2W4MR4HARWJOBDF4Q, seg_pos={0 0}, upload=473908]: Uploading piece...
Dec 28 03:51:59.655943527 (540h22m59.527026562s) [noise=true , address=45.91.116.14:4190 , piece_id=VX56NISLUVDY7BDGSZPHWZCZMMQEQAFIGCS4PXEQZISU7GDHMOUA, seg_pos={0 0}, upload=473908]: Uploading piece...
Dec 28 03:51:59.656053106 (540h22m59.527136141s) [noise=true , address=145.79.227.14:4838 , piece_id=6W4FBPQD2NIWLTKDIMLEBLIFGA5JIOA45W5YGHOU3AOZOJMFQCAA, seg_pos={0 0}, upload=473908]: Uploading piece...
Dec 28 03:51:59.656311858 (540h22m59.527394893s) [noise=true , address=145.79.227.10:4402 , piece_id=DZM5PJ3GIPT7OIW7KW5K7GOKLSFMJSJWVRHBTUK3L6GSSQ4ZMZ7Q, seg_pos={0 0}, upload=473908]: Uploading piece...
Dec 28 03:51:59.656585389 (540h22m59.527668434s) [noise=true , address=45.91.116.10:4082 , piece_id=XDBMTSXPQVCE7K2OUJMX7HCO2CRIO2JELT4YIFHEVL4XOSDIN4ZQ, seg_pos={0 0}, upload=473908]: Uploading piece...
Dec 28 03:51:59.656899217 (540h22m59.527982252s) [noise=true , address=45.91.116.9:4263 , piece_id=ICXTWYHGRYAOWYSSN4BPPPGYMVGWERQJ4D2UPSWKJKLO7BXITCTA, seg_pos={0 0}, upload=473908]: Uploading piece...
Dec 28 03:51:59.657036458 (540h22m59.528119503s) [noise=true , address=45.91.116.6:4976 , piece_id=EUTRPUS62FFJXQ4MGKU34HKB6SZXLZYZO2HMFIXFU35AFFK22ECQ, seg_pos={0 0}, upload=473908]: Uploading piece...
Dec 28 03:51:59.657406633 (540h22m59.528489678s) [noise=true , address=145.79.227.50:4373 , piece_id=7KL3OZ3HSSME3U4VO7E7O3CED7SWYZUPUX6DXAA2KPMIH4EFNH7Q, seg_pos={0 0}, upload=473908]: Uploading piece...
Dec 28 03:51:59.657761158 (540h22m59.528844204s) [noise=true , address=145.79.227.13:4803 , piece_id=E5233Z5PRSOZ7VWJJZHKP7TDB2YZ5GDXU47XWTWPFUS7U4BUHY5A, seg_pos={0 0}, upload=473908]: Uploading piece...
Dec 28 03:51:59.658140540 (540h22m59.529223585s) [noise=true , address=145.79.227.15:4873 , piece_id=ZU2FORZXRC67VKDYEGB3JURNCRO5ZKH3WRHLUUWHFRMTHSZQGD5Q, seg_pos={0 0}, upload=473908]: Uploading piece...
Dec 28 03:51:59.658171329 (540h22m59.529254374s) [noise=true , address=45.91.116.12:4228 , piece_id=3SFQIZV6KAIGUF4DEPCJ2OKEAOXHDE6E7MGB4BOYRXL6OSSNJ5OQ, seg_pos={0 0}, upload=473908]: Uploading piece...
Dec 28 03:52:01.384030301 (540h23m1.255113336s) [upload=473908]: Next returned nil segment
Dec 28 03:52:01.384045119 (540h23m1.255128164s) [upload=473908]: Waiting for error group managing segments...
Dec 28 03:52:01.666973103 (540h23m1.538056148s) [noise=true , address=145.79.227.50:4373 , piece_id=7KL3OZ3HSSME3U4VO7E7O3CED7SWYZUPUX6DXAA2KPMIH4EFNH7Q, seg_pos={0 0}, upload=473908]: Done uploading piece. err: <nil>
Dec 28 03:52:01.690885526 (540h23m1.561968571s) [noise=true , address=145.79.227.10:4402 , piece_id=DZM5PJ3GIPT7OIW7KW5K7GOKLSFMJSJWVRHBTUK3L6GSSQ4ZMZ7Q, seg_pos={0 0}, upload=473908]: Done uploading piece. err: <nil>
Dec 28 03:52:01.691361382 (540h23m1.562444427s) [noise=true , address=145.79.227.15:4873 , piece_id=ZU2FORZXRC67VKDYEGB3JURNCRO5ZKH3WRHLUUWHFRMTHSZQGD5Q, seg_pos={0 0}, upload=473908]: Done uploading piece. err: <nil>
Dec 28 03:52:01.691848750 (540h23m1.562931795s) [noise=true , address=145.79.227.13:4803 , piece_id=E5233Z5PRSOZ7VWJJZHKP7TDB2YZ5GDXU47XWTWPFUS7U4BUHY5A, seg_pos={0 0}, upload=473908]: Done uploading piece. err: <nil>
Dec 28 03:52:01.697869995 (540h23m1.56895304s) [noise=true , address=145.79.227.39:4014 , piece_id=56263GZBKICQ652BAVE6YO4RY23W3L6BBVR2W4MR4HARWJOBDF4Q, seg_pos={0 0}, upload=473908]: Done uploading piece. err: <nil>
Dec 28 03:52:01.729400635 (540h23m1.60048368s) [noise=true , address=145.79.227.2:4115 , piece_id=ZDZ4LWRF3C2PWGUG67CN2URYX3YCE6KUP5MUGO3335AFEWQJK3WA, seg_pos={0 0}, upload=473908]: Done uploading piece. err: <nil>
Dec 28 03:52:01.749728844 (540h23m1.620811889s) [noise=true , address=145.79.227.14:4838 , piece_id=6W4FBPQD2NIWLTKDIMLEBLIFGA5JIOA45W5YGHOU3AOZOJMFQCAA, seg_pos={0 0}, upload=473908]: Done uploading piece. err: <nil>
Dec 28 03:52:01.978514911 (540h23m1.849597947s) [noise=true , address=45.91.116.12:4228 , piece_id=3SFQIZV6KAIGUF4DEPCJ2OKEAOXHDE6E7MGB4BOYRXL6OSSNJ5OQ, seg_pos={0 0}, upload=473908]: Done uploading piece. err: <nil>
Dec 28 03:52:02.007330220 (540h23m1.878413265s) [noise=true , address=45.91.116.9:4263 , piece_id=ICXTWYHGRYAOWYSSN4BPPPGYMVGWERQJ4D2UPSWKJKLO7BXITCTA, seg_pos={0 0}, upload=473908]: Done uploading piece. err: <nil>
Dec 28 03:52:02.291096190 (540h23m2.162179235s) [noise=true , address=77.90.31.19:4842 , piece_id=YH224UFOU6AFIDGZNEAYG4JQYOWGWPX3PIQJU25GOEC6UB5S4XNQ, seg_pos={0 0}, upload=473908]: Done uploading piece. err: <nil>
Dec 28 03:52:02.401832711 (540h23m2.272915756s) [noise=true , address=45.91.116.7:4011 , piece_id=QW35GCUQ5KJBLUKR7ZUW3PUK44YIJVBMYAHIXSGUGDPCHJQCCZUA, seg_pos={0 0}, upload=473908]: Done uploading piece. err: <nil>
Dec 28 03:52:02.602916329 (540h23m2.473999374s) [noise=true , address=45.91.116.5:4300 , piece_id=LNYOIQ3J6MI4Z4FUAZJTLSI7LFC3U5ZF3YHAASMP2TH4TO3ANDSQ, seg_pos={0 0}, upload=473908]: Done uploading piece. err: <nil>
Dec 28 03:52:02.703592313 (540h23m2.574675358s) [noise=true , address=45.91.116.14:4190 , piece_id=VX56NISLUVDY7BDGSZPHWZCZMMQEQAFIGCS4PXEQZISU7GDHMOUA, seg_pos={0 0}, upload=473908]: Done uploading piece. err: <nil>
Dec 28 03:52:02.703621258 (540h23m2.574704293s) [seg_pos={0 0}, upload=473908]: Segment reached optimal threshold of 13 pieces.
Dec 28 03:52:02.704004318 (540h23m2.575087373s) [noise=true , address=45.91.116.10:4082 , piece_id=XDBMTSXPQVCE7K2OUJMX7HCO2CRIO2JELT4YIFHEVL4XOSDIN4ZQ, seg_pos={0 0}, upload=473908]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF
Dec 28 03:52:02.704014837 (540h23m2.575097882s) [noise=true , address=45.91.116.10:4082 , piece_id=XDBMTSXPQVCE7K2OUJMX7HCO2CRIO2JELT4YIFHEVL4XOSDIN4ZQ, seg_pos={0 0}, upload=473908]: Long tail cancellation due to optimal threshold
Dec 28 03:52:02.704148482 (540h23m2.575231547s) [noise=true , address=45.91.116.13:4156 , piece_id=P4IVKHNZDQNNRBTLE4AO6DRO5NEGVHW6K3AYJ37FTEICYVVB5E4A, seg_pos={0 0}, upload=473908]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF
Dec 28 03:52:02.704191013 (540h23m2.575274068s) [noise=true , address=45.91.116.13:4156 , piece_id=P4IVKHNZDQNNRBTLE4AO6DRO5NEGVHW6K3AYJ37FTEICYVVB5E4A, seg_pos={0 0}, upload=473908]: Long tail cancellation due to optimal threshold
Dec 28 03:52:02.704463141 (540h23m2.575546186s) [noise=true , address=45.91.116.6:4976 , piece_id=EUTRPUS62FFJXQ4MGKU34HKB6SZXLZYZO2HMFIXFU35AFFK22ECQ, seg_pos={0 0}, upload=473908]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF
Dec 28 03:52:02.704472759 (540h23m2.575555794s) [noise=true , address=45.91.116.6:4976 , piece_id=EUTRPUS62FFJXQ4MGKU34HKB6SZXLZYZO2HMFIXFU35AFFK22ECQ, seg_pos={0 0}, upload=473908]: Long tail cancellation due to optimal threshold
Dec 28 03:52:02.704493138 (540h23m2.575576173s) [seg_pos={0 0}, upload=473908]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 28 03:52:02.704607756 (540h23m2.575690801s) [upload=473908]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 03:52:02.704617645 (540h23m2.57570069s) [upload=473908]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 03:52:03.001299398 (540h23m2.872382474s) [upload=473908]: Done uploading segments...
Dec 28 03:52:03.001326409 (540h23m2.872409464s) [upload=473908]: Upload finished. err: <nil>
Duration: 6.112808956s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [............................................X...............] duration:
15.695720318s trace:
https://jaeger.dev.storj.io/trace/3b8130f32e60318d
FAIL (2025-12-28 03:52:30 UTC)
orbiter warning: context deadline exceeded
Duration: 15.695749553s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
51 out of 60 is failed (threshold=3) [..XX..XXXXXX.XXXXXXXXXXXXXXXXX.XXXXX.XXXXXXXXXXXXXXXXX..XXXX] duration:
30.002217506s trace:
https://jaeger.dev.storj.io/trace/1f754c262e8a6f59
FAIL (2025-12-28 03:52:28 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA, Piece ID: 3SFQIZV6KAIGUF4DEPCJ2OKEAOXHDE6E7MGB4BOYRXL6OSSNJ5OQ) context deadline exceeded; piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: QW35GCUQ5KJBLUKR7ZUW3PUK44YIJVBMYAHIXSGUGDPCHJQCCZUA) context deadline exceeded
======LOGS======
Dec 28 03:51:59.409570400 (540h22m59.280653445s) [upload=473908]: Starting upload
Dec 28 03:51:59.409624112 (540h22m59.280707157s) [upload=473908]: Done starting upload
Dec 28 03:51:59.410221440 (540h22m59.281304475s) [upload=473908]: Uploading segments...
Dec 28 03:51:59.410295541 (540h22m59.281378576s) [upload=473908]: Got next segment. Inline: false
Dec 28 03:51:59.410427493 (540h22m59.281510538s) [upload=473908]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 28 03:51:59.410432622 (540h22m59.281515667s) [upload=473908]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 28 03:51:59.654512031 (540h22m59.525595066s) [seg_pos={0 0}, upload=473908]: Begin upload segment...
Dec 28 03:51:59.654571474 (540h22m59.525654509s) [seg_pos={0 0}, upload=473908]: Done begin upload segment.
Dec 28 03:51:59.654674801 (540h22m59.525757846s) [noise=true , address=45.91.116.7:4011 , piece_id=QW35GCUQ5KJBLUKR7ZUW3PUK44YIJVBMYAHIXSGUGDPCHJQCCZUA, seg_pos={0 0}, upload=473908]: Uploading piece...
Dec 28 03:51:59.654679921 (540h22m59.525763006s) [noise=true , address=45.91.116.13:4156 , piece_id=P4IVKHNZDQNNRBTLE4AO6DRO5NEGVHW6K3AYJ37FTEICYVVB5E4A, seg_pos={0 0}, upload=473908]: Uploading piece...
Dec 28 03:51:59.655256279 (540h22m59.526339314s) [noise=true , address=145.79.227.2:4115 , piece_id=ZDZ4LWRF3C2PWGUG67CN2URYX3YCE6KUP5MUGO3335AFEWQJK3WA, seg_pos={0 0}, upload=473908]: Uploading piece...
Dec 28 03:51:59.655482629 (540h22m59.526565664s) [noise=true , address=77.90.31.19:4842 , piece_id=YH224UFOU6AFIDGZNEAYG4JQYOWGWPX3PIQJU25GOEC6UB5S4XNQ, seg_pos={0 0}, upload=473908]: Uploading piece...
Dec 28 03:51:59.655614020 (540h22m59.526697065s) [noise=true , address=45.91.116.5:4300 , piece_id=LNYOIQ3J6MI4Z4FUAZJTLSI7LFC3U5ZF3YHAASMP2TH4TO3ANDSQ, seg_pos={0 0}, upload=473908]: Uploading piece...
Dec 28 03:51:59.655776630 (540h22m59.526859675s) [noise=true , address=145.79.227.39:4014 , piece_id=56263GZBKICQ652BAVE6YO4RY23W3L6BBVR2W4MR4HARWJOBDF4Q, seg_pos={0 0}, upload=473908]: Uploading piece...
Dec 28 03:51:59.655943527 (540h22m59.527026562s) [noise=true , address=45.91.116.14:4190 , piece_id=VX56NISLUVDY7BDGSZPHWZCZMMQEQAFIGCS4PXEQZISU7GDHMOUA, seg_pos={0 0}, upload=473908]: Uploading piece...
Dec 28 03:51:59.656053106 (540h22m59.527136141s) [noise=true , address=145.79.227.14:4838 , piece_id=6W4FBPQD2NIWLTKDIMLEBLIFGA5JIOA45W5YGHOU3AOZOJMFQCAA, seg_pos={0 0}, upload=473908]: Uploading piece...
Dec 28 03:51:59.656311858 (540h22m59.527394893s) [noise=true , address=145.79.227.10:4402 , piece_id=DZM5PJ3GIPT7OIW7KW5K7GOKLSFMJSJWVRHBTUK3L6GSSQ4ZMZ7Q, seg_pos={0 0}, upload=473908]: Uploading piece...
Dec 28 03:51:59.656585389 (540h22m59.527668434s) [noise=true , address=45.91.116.10:4082 , piece_id=XDBMTSXPQVCE7K2OUJMX7HCO2CRIO2JELT4YIFHEVL4XOSDIN4ZQ, seg_pos={0 0}, upload=473908]: Uploading piece...
Dec 28 03:51:59.656899217 (540h22m59.527982252s) [noise=true , address=45.91.116.9:4263 , piece_id=ICXTWYHGRYAOWYSSN4BPPPGYMVGWERQJ4D2UPSWKJKLO7BXITCTA, seg_pos={0 0}, upload=473908]: Uploading piece...
Dec 28 03:51:59.657036458 (540h22m59.528119503s) [noise=true , address=45.91.116.6:4976 , piece_id=EUTRPUS62FFJXQ4MGKU34HKB6SZXLZYZO2HMFIXFU35AFFK22ECQ, seg_pos={0 0}, upload=473908]: Uploading piece...
Dec 28 03:51:59.657406633 (540h22m59.528489678s) [noise=true , address=145.79.227.50:4373 , piece_id=7KL3OZ3HSSME3U4VO7E7O3CED7SWYZUPUX6DXAA2KPMIH4EFNH7Q, seg_pos={0 0}, upload=473908]: Uploading piece...
Dec 28 03:51:59.657761158 (540h22m59.528844204s) [noise=true , address=145.79.227.13:4803 , piece_id=E5233Z5PRSOZ7VWJJZHKP7TDB2YZ5GDXU47XWTWPFUS7U4BUHY5A, seg_pos={0 0}, upload=473908]: Uploading piece...
Dec 28 03:51:59.658140540 (540h22m59.529223585s) [noise=true , address=145.79.227.15:4873 , piece_id=ZU2FORZXRC67VKDYEGB3JURNCRO5ZKH3WRHLUUWHFRMTHSZQGD5Q, seg_pos={0 0}, upload=473908]: Uploading piece...
Dec 28 03:51:59.658171329 (540h22m59.529254374s) [noise=true , address=45.91.116.12:4228 , piece_id=3SFQIZV6KAIGUF4DEPCJ2OKEAOXHDE6E7MGB4BOYRXL6OSSNJ5OQ, seg_pos={0 0}, upload=473908]: Uploading piece...
Dec 28 03:52:01.384030301 (540h23m1.255113336s) [upload=473908]: Next returned nil segment
Dec 28 03:52:01.384045119 (540h23m1.255128164s) [upload=473908]: Waiting for error group managing segments...
Dec 28 03:52:01.666973103 (540h23m1.538056148s) [noise=true , address=145.79.227.50:4373 , piece_id=7KL3OZ3HSSME3U4VO7E7O3CED7SWYZUPUX6DXAA2KPMIH4EFNH7Q, seg_pos={0 0}, upload=473908]: Done uploading piece. err: <nil>
Dec 28 03:52:01.690885526 (540h23m1.561968571s) [noise=true , address=145.79.227.10:4402 , piece_id=DZM5PJ3GIPT7OIW7KW5K7GOKLSFMJSJWVRHBTUK3L6GSSQ4ZMZ7Q, seg_pos={0 0}, upload=473908]: Done uploading piece. err: <nil>
Dec 28 03:52:01.691361382 (540h23m1.562444427s) [noise=true , address=145.79.227.15:4873 , piece_id=ZU2FORZXRC67VKDYEGB3JURNCRO5ZKH3WRHLUUWHFRMTHSZQGD5Q, seg_pos={0 0}, upload=473908]: Done uploading piece. err: <nil>
Dec 28 03:52:01.691848750 (540h23m1.562931795s) [noise=true , address=145.79.227.13:4803 , piece_id=E5233Z5PRSOZ7VWJJZHKP7TDB2YZ5GDXU47XWTWPFUS7U4BUHY5A, seg_pos={0 0}, upload=473908]: Done uploading piece. err: <nil>
Dec 28 03:52:01.697869995 (540h23m1.56895304s) [noise=true , address=145.79.227.39:4014 , piece_id=56263GZBKICQ652BAVE6YO4RY23W3L6BBVR2W4MR4HARWJOBDF4Q, seg_pos={0 0}, upload=473908]: Done uploading piece. err: <nil>
Dec 28 03:52:01.729400635 (540h23m1.60048368s) [noise=true , address=145.79.227.2:4115 , piece_id=ZDZ4LWRF3C2PWGUG67CN2URYX3YCE6KUP5MUGO3335AFEWQJK3WA, seg_pos={0 0}, upload=473908]: Done uploading piece. err: <nil>
Dec 28 03:52:01.749728844 (540h23m1.620811889s) [noise=true , address=145.79.227.14:4838 , piece_id=6W4FBPQD2NIWLTKDIMLEBLIFGA5JIOA45W5YGHOU3AOZOJMFQCAA, seg_pos={0 0}, upload=473908]: Done uploading piece. err: <nil>
Dec 28 03:52:01.978514911 (540h23m1.849597947s) [noise=true , address=45.91.116.12:4228 , piece_id=3SFQIZV6KAIGUF4DEPCJ2OKEAOXHDE6E7MGB4BOYRXL6OSSNJ5OQ, seg_pos={0 0}, upload=473908]: Done uploading piece. err: <nil>
Dec 28 03:52:02.007330220 (540h23m1.878413265s) [noise=true , address=45.91.116.9:4263 , piece_id=ICXTWYHGRYAOWYSSN4BPPPGYMVGWERQJ4D2UPSWKJKLO7BXITCTA, seg_pos={0 0}, upload=473908]: Done uploading piece. err: <nil>
Dec 28 03:52:02.291096190 (540h23m2.162179235s) [noise=true , address=77.90.31.19:4842 , piece_id=YH224UFOU6AFIDGZNEAYG4JQYOWGWPX3PIQJU25GOEC6UB5S4XNQ, seg_pos={0 0}, upload=473908]: Done uploading piece. err: <nil>
Dec 28 03:52:02.401832711 (540h23m2.272915756s) [noise=true , address=45.91.116.7:4011 , piece_id=QW35GCUQ5KJBLUKR7ZUW3PUK44YIJVBMYAHIXSGUGDPCHJQCCZUA, seg_pos={0 0}, upload=473908]: Done uploading piece. err: <nil>
Dec 28 03:52:02.602916329 (540h23m2.473999374s) [noise=true , address=45.91.116.5:4300 , piece_id=LNYOIQ3J6MI4Z4FUAZJTLSI7LFC3U5ZF3YHAASMP2TH4TO3ANDSQ, seg_pos={0 0}, upload=473908]: Done uploading piece. err: <nil>
Dec 28 03:52:02.703592313 (540h23m2.574675358s) [noise=true , address=45.91.116.14:4190 , piece_id=VX56NISLUVDY7BDGSZPHWZCZMMQEQAFIGCS4PXEQZISU7GDHMOUA, seg_pos={0 0}, upload=473908]: Done uploading piece. err: <nil>
Dec 28 03:52:02.703621258 (540h23m2.574704293s) [seg_pos={0 0}, upload=473908]: Segment reached optimal threshold of 13 pieces.
Dec 28 03:52:02.704004318 (540h23m2.575087373s) [noise=true , address=45.91.116.10:4082 , piece_id=XDBMTSXPQVCE7K2OUJMX7HCO2CRIO2JELT4YIFHEVL4XOSDIN4ZQ, seg_pos={0 0}, upload=473908]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF
Dec 28 03:52:02.704014837 (540h23m2.575097882s) [noise=true , address=45.91.116.10:4082 , piece_id=XDBMTSXPQVCE7K2OUJMX7HCO2CRIO2JELT4YIFHEVL4XOSDIN4ZQ, seg_pos={0 0}, upload=473908]: Long tail cancellation due to optimal threshold
Dec 28 03:52:02.704148482 (540h23m2.575231547s) [noise=true , address=45.91.116.13:4156 , piece_id=P4IVKHNZDQNNRBTLE4AO6DRO5NEGVHW6K3AYJ37FTEICYVVB5E4A, seg_pos={0 0}, upload=473908]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF
Dec 28 03:52:02.704191013 (540h23m2.575274068s) [noise=true , address=45.91.116.13:4156 , piece_id=P4IVKHNZDQNNRBTLE4AO6DRO5NEGVHW6K3AYJ37FTEICYVVB5E4A, seg_pos={0 0}, upload=473908]: Long tail cancellation due to optimal threshold
Dec 28 03:52:02.704463141 (540h23m2.575546186s) [noise=true , address=45.91.116.6:4976 , piece_id=EUTRPUS62FFJXQ4MGKU34HKB6SZXLZYZO2HMFIXFU35AFFK22ECQ, seg_pos={0 0}, upload=473908]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF
Dec 28 03:52:02.704472759 (540h23m2.575555794s) [noise=true , address=45.91.116.6:4976 , piece_id=EUTRPUS62FFJXQ4MGKU34HKB6SZXLZYZO2HMFIXFU35AFFK22ECQ, seg_pos={0 0}, upload=473908]: Long tail cancellation due to optimal threshold
Dec 28 03:52:02.704493138 (540h23m2.575576173s) [seg_pos={0 0}, upload=473908]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 28 03:52:02.704607756 (540h23m2.575690801s) [upload=473908]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 03:52:02.704617645 (540h23m2.57570069s) [upload=473908]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 03:52:03.001299398 (540h23m2.872382474s) [upload=473908]: Done uploading segments...
Dec 28 03:52:03.001326409 (540h23m2.872409464s) [upload=473908]: Upload finished. err: <nil>
Duration: 30.002239467s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [.....................X...X..................................] duration:
43.245647092s trace:
https://jaeger.dev.storj.io/trace/54422f75ff432c0
FAIL (2025-12-28 03:52:22 UTC)
orbiter warning:
operation: DeleteObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
, piece_id=SPQSUVHIHI32ZMIRE4NHSNJCMWS4NS6ZZPYF6FV7FQH6VADQIV5A, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:35.720215866 (539h54m35.591298911s) [noise=true , address=145.79.227.43:4187 , piece_id=MRILXYFSWVSH35ZCCFZWLEXENWJII6G6LOR2N277GIBMI4PXPPTQ, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:35.720567665 (539h54m35.5916507s) [noise=true , address=145.79.227.33:4896 , piece_id=7NRM7I4IDHW2WGOCG3B6L6HN74ANXWDJCFRPBM4BCHWJELSNNUXA, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:35.720883607 (539h54m35.591966652s) [noise=true , address=145.79.227.40:4062 , piece_id=RCVKQLDSWIWX2FQBTUFPEUXQZLP2C3ZCLV2JWHOV7CMJ7ZBNWZNQ, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:35.721874904 (539h54m35.592957949s) [noise=true , address=64.49.13.140:4513 , piece_id=Y3HW5I2FX2CWSPVUS7CU3JPNN2RHP2ASL757AUYINBKRKCLXAJGQ, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:35.722249658 (539h54m35.593332693s) [noise=true , address=64.49.13.169:4715 , piece_id=FRVNWFP2AE4DXE4RKZRE6WTK43T74BTTYM2U7E4P3B4XZIMC2QMQ, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:35.722786881 (539h54m35.593869926s) [noise=true , address=77.90.31.14:4673 , piece_id=ZIB5VOFL7PJSRFDV3HP5XAKYHOJXRVHCYDPIHKRYPNM5ROAY5DTQ, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:35.723112100 (539h54m35.594195135s) [noise=true , address=77.90.31.17:4779 , piece_id=TLU5ZLZRVAMGQLSE2LLNIUE4WOWJBMFJGFQKC3KOUJ23V3UVFMRA, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:35.723440756 (539h54m35.594523791s) [noise=true , address=145.79.227.37:4569 , piece_id=DUXSQUT36XUF66ZVQ2V7ZWNXA6JZKBDO5RXT3QVT26GE2FTH3QXQ, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:35.723849544 (539h54m35.594932589s) [noise=true , address=145.79.227.27:4723 , piece_id=4DECSGDNZLXNQVPU7QALLPFQYDO7MOG6TIBNWQ5UHNVCIHB4GUYQ, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:35.724241290 (539h54m35.595324335s) [noise=true , address=77.90.31.11:4587 , piece_id=6467VZCLWHJTZ36ENMEOEOU2E4MAUCDL2XHNAOMJTCCTUSN33NFA, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:35.724607808 (539h54m35.595690854s) [noise=true , address=145.79.227.47:4272 , piece_id=FL2U4W2YCV5ANDVFHB7H4TFT3SYTBGNLI3KZDI7MY2K6NTSMRUAA, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:35.724974035 (539h54m35.59605707s) [noise=true , address=145.79.227.18:4040 , piece_id=W55GK3QDHRHNR6YJ7JKVZMIEKFQLZSTXKSQ5OMW3U7XRI44EM6RA, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:35.725302281 (539h54m35.596385316s) [noise=true , address=145.79.227.38:4989 , piece_id=OUINZ6BXOLFCDNCXMX7PPFSKWMJ7IPZWO7DAGLGVV2AB7UHO3JYQ, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:35.726824420 (539h54m35.597907455s) [noise=true , address=145.79.227.26:4669 , piece_id=5D7SGVO274FEZ4JVESTMUYQ3RZ3DJA5DL2QZV5YKMXEB4TM6PHAA, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:37.063163473 (539h54m36.934246518s) [upload=473490]: Next returned nil segment
Dec 28 03:23:37.063176499 (539h54m36.934259534s) [upload=473490]: Waiting for error group managing segments...
Dec 28 03:23:37.416516311 (539h54m37.287599376s) [noise=true , address=64.49.13.140:4513 , piece_id=Y3HW5I2FX2CWSPVUS7CU3JPNN2RHP2ASL757AUYINBKRKCLXAJGQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil>
Dec 28 03:23:37.606629586 (539h54m37.477712622s) [noise=true , address=145.79.227.27:4723 , piece_id=4DECSGDNZLXNQVPU7QALLPFQYDO7MOG6TIBNWQ5UHNVCIHB4GUYQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil>
Dec 28 03:23:37.617730204 (539h54m37.488813249s) [noise=true , address=145.79.227.26:4669 , piece_id=5D7SGVO274FEZ4JVESTMUYQ3RZ3DJA5DL2QZV5YKMXEB4TM6PHAA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil>
Dec 28 03:23:37.632542950 (539h54m37.503625995s) [noise=true , address=145.79.227.38:4989 , piece_id=OUINZ6BXOLFCDNCXMX7PPFSKWMJ7IPZWO7DAGLGVV2AB7UHO3JYQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil>
Dec 28 03:23:37.643987531 (539h54m37.515070566s) [noise=true , address=145.79.227.47:4272 , piece_id=FL2U4W2YCV5ANDVFHB7H4TFT3SYTBGNLI3KZDI7MY2K6NTSMRUAA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil>
Dec 28 03:23:37.652913619 (539h54m37.523996654s) [noise=true , address=145.79.227.18:4040 , piece_id=W55GK3QDHRHNR6YJ7JKVZMIEKFQLZSTXKSQ5OMW3U7XRI44EM6RA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil>
Dec 28 03:23:37.661966357 (539h54m37.533049392s) [noise=true , address=145.79.227.37:4569 , piece_id=DUXSQUT36XUF66ZVQ2V7ZWNXA6JZKBDO5RXT3QVT26GE2FTH3QXQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil>
Dec 28 03:23:37.778627848 (539h54m37.649710893s) [noise=true , address=77.90.31.11:4561 , piece_id=HIUCZRRZOMU4RBYZU5SADJAOS4RVML2I6PILIX2V65RS22EBN42Q, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil>
Dec 28 03:23:37.815974812 (539h54m37.687057857s) [noise=true , address=45.91.116.14:4217 , piece_id=SPQSUVHIHI32ZMIRE4NHSNJCMWS4NS6ZZPYF6FV7FQH6VADQIV5A, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil>
Dec 28 03:23:37.861731780 (539h54m37.732814815s) [noise=true , address=77.90.31.14:4673 , piece_id=ZIB5VOFL7PJSRFDV3HP5XAKYHOJXRVHCYDPIHKRYPNM5ROAY5DTQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil>
Dec 28 03:23:38.093273211 (539h54m37.964356256s) [noise=true , address=77.90.31.11:4587 , piece_id=6467VZCLWHJTZ36ENMEOEOU2E4MAUCDL2XHNAOMJTCCTUSN33NFA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil>
Dec 28 03:23:38.267694395 (539h54m38.13877745s) [noise=true , address=77.90.31.17:4779 , piece_id=TLU5ZLZRVAMGQLSE2LLNIUE4WOWJBMFJGFQKC3KOUJ23V3UVFMRA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil>
Dec 28 03:23:38.540947761 (539h54m38.412030796s) [noise=true , address=145.79.227.40:4062 , piece_id=RCVKQLDSWIWX2FQBTUFPEUXQZLP2C3ZCLV2JWHOV7CMJ7ZBNWZNQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil>
Dec 28 03:23:38.815393527 (539h54m38.686476562s) [noise=true , address=145.79.227.43:4187 , piece_id=MRILXYFSWVSH35ZCCFZWLEXENWJII6G6LOR2N277GIBMI4PXPPTQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil>
Dec 28 03:23:38.815412894 (539h54m38.686495939s) [seg_pos={0 0}, upload=473490]: Segment reached optimal threshold of 14 pieces.
Dec 28 03:23:38.815594951 (539h54m38.686677986s) [noise=true , address=64.49.13.169:4715 , piece_id=FRVNWFP2AE4DXE4RKZRE6WTK43T74BTTYM2U7E4P3B4XZIMC2QMQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:43502->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:43502->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43502->64.49.13.169:4715: use of closed network connection
Dec 28 03:23:38.815601032 (539h54m38.686684077s) [noise=true , address=64.49.13.169:4715 , piece_id=FRVNWFP2AE4DXE4RKZRE6WTK43T74BTTYM2U7E4P3B4XZIMC2QMQ, seg_pos={0 0}, upload=473490]: Long tail cancellation due to optimal threshold
Dec 28 03:23:38.815826983 (539h54m38.686910018s) [noise=true , address=145.79.227.41:4118 , piece_id=C7JJ3NHR5355M64G2EF2WWERQVGWR243RQHDV3GBAIC3A66JDMHQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128w37UHxJB9R49iwmHbATt3bXUb1FMTQJ3Wjka7So7eqKWg4iF): protocol: expected piece hash; context canceled; EOF
Dec 28 03:23:38.815834507 (539h54m38.686917542s) [noise=true , address=145.79.227.41:4118 , piece_id=C7JJ3NHR5355M64G2EF2WWERQVGWR243RQHDV3GBAIC3A66JDMHQ, seg_pos={0 0}, upload=473490]: Long tail cancellation due to optimal threshold
Dec 28 03:23:38.815890433 (539h54m38.686973468s) [noise=true , address=145.79.227.33:4896 , piece_id=7NRM7I4IDHW2WGOCG3B6L6HN74ANXWDJCFRPBM4BCHWJELSNNUXA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ndSgjgfZMVQni5tw1ugdTrhJt4c4NagdvgYTbSbjatYYtfu1o): protocol: expected piece hash; context canceled; EOF
Dec 28 03:23:38.815914619 (539h54m38.686997654s) [noise=true , address=145.79.227.33:4896 , piece_id=7NRM7I4IDHW2WGOCG3B6L6HN74ANXWDJCFRPBM4BCHWJELSNNUXA, seg_pos={0 0}, upload=473490]: Long tail cancellation due to optimal threshold
Dec 28 03:23:38.816105613 (539h54m38.687188648s) [noise=true , address=145.79.227.28:4731 , piece_id=EPW3LUA3EWRCURHZ66NCRCA36CNIW4GYQMFNKPZZTRCB3YBGTKJA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NQzTwVYQa6ZzxAHQm1apkwWnqJstTnAJQyHKhBzC2jTN9kEVK): protocol: write tcp 188.166.225.170:58940->145.79.227.28:4731: use of closed network connection; write tcp 188.166.225.170:58940->145.79.227.28:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58940->145.79.227.28:4731: use of closed network connection
Dec 28 03:23:38.816112426 (539h54m38.687195461s) [noise=true , address=145.79.227.28:4731 , piece_id=EPW3LUA3EWRCURHZ66NCRCA36CNIW4GYQMFNKPZZTRCB3YBGTKJA, seg_pos={0 0}, upload=473490]: Long tail cancellation due to optimal threshold
Dec 28 03:23:38.816128436 (539h54m38.687211481s) [seg_pos={0 0}, upload=473490]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 28 03:23:38.816257582 (539h54m38.687340627s) [upload=473490]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 03:23:38.816265778 (539h54m38.687348813s) [upload=473490]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 03:23:39.111115407 (539h54m38.982198442s) [upload=473490]: Done uploading segments...
Dec 28 03:23:39.111138972 (539h54m38.982222008s) [upload=473490]: Upload finished. err: <nil>
Duration: 43.245676368s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
51 out of 60 is failed (threshold=3) [X..XX..XXXXXX.XXXXXXXXXXXXXXXXX.XXXXX.XXXXXXXXXXXXXXXXX..XXX] duration:
30.003304629s trace:
https://jaeger.dev.storj.io/trace/381fa53936846526
FAIL (2025-12-28 03:51:56 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
8 (540h22m27.217443303s) [upload=473901]: Done starting upload
Dec 28 03:51:27.346557854 (540h22m27.217640889s) [upload=473901]: Uploading segments...
Dec 28 03:51:27.346763305 (540h22m27.21784634s) [upload=473901]: Got next segment. Inline: false
Dec 28 03:51:27.346801908 (540h22m27.217884943s) [upload=473901]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 28 03:51:27.346807268 (540h22m27.217890303s) [upload=473901]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 28 03:51:27.592436829 (540h22m27.463519864s) [seg_pos={0 0}, upload=473901]: Begin upload segment...
Dec 28 03:51:27.592484500 (540h22m27.463567535s) [seg_pos={0 0}, upload=473901]: Done begin upload segment.
Dec 28 03:51:27.592504227 (540h22m27.463587262s) [noise=true , address=145.79.227.22:4511 , piece_id=DOMEZRCPJLN2FPXWCNA2FSAAXSTNKKJJHWHKXLLT7AZWSSYPJSRQ, seg_pos={0 0}, upload=473901]: Uploading piece...
Dec 28 03:51:27.592517432 (540h22m27.463600468s) [noise=true , address=45.91.116.14:4191 , piece_id=HT2V76RPE45TU4O5CB3VMCWQWYSUJKBG3LSNH2WWHI6HTD3D423A, seg_pos={0 0}, upload=473901]: Uploading piece...
Dec 28 03:51:27.593031792 (540h22m27.464114827s) [noise=true , address=145.79.227.3:4150 , piece_id=FIFEQJOEILOERQQUANU277W3CLXXSMYWIQGGZ6EUK4EMOSCDDB7Q, seg_pos={0 0}, upload=473901]: Uploading piece...
Dec 28 03:51:27.593463674 (540h22m27.464546719s) [noise=true , address=77.90.31.21:4880 , piece_id=6PHUE2QSDNDT3OR4QZLTUJAOCZSU4WWW64WMM74YAOYVTQGIMEQA, seg_pos={0 0}, upload=473901]: Uploading piece...
Dec 28 03:51:27.593774867 (540h22m27.464857922s) [noise=true , address=45.91.116.4:4937 , piece_id=HU3AX6MJ4PEL3XZXYN2YED7RHZTCWIN2CCUUQGY4DC5ENIZDQS4A, seg_pos={0 0}, upload=473901]: Uploading piece...
Dec 28 03:51:27.594145583 (540h22m27.465228618s) [noise=true , address=145.79.227.18:4006 , piece_id=WTBMCGGZDWOSGJ6GY54BV6L3CZTGL7SZMDBNFIMRCKTPSAUJRGVQ, seg_pos={0 0}, upload=473901]: Uploading piece...
Dec 28 03:51:27.594557296 (540h22m27.465640331s) [noise=true , address=145.79.227.50:4374 , piece_id=AP2UXSDPJEKO6YHNT46GKQGYWYX3MRB6L7I6X7YCANWQBROZALGA, seg_pos={0 0}, upload=473901]: Uploading piece...
Dec 28 03:51:27.595190172 (540h22m27.466273217s) [noise=true , address=45.91.116.8:4045 , piece_id=QTRX4VGVPE6CWXSUO2UW22HWABMV72JLUM7TGBJTVPRALPQCSC6A, seg_pos={0 0}, upload=473901]: Uploading piece...
Dec 28 03:51:27.595799432 (540h22m27.466882467s) [noise=true , address=145.79.227.51:4410 , piece_id=SP6N5RLRL7EW6UTOBF74I2VROC3VEPJN6NK5KNCSZE6OHLHKN6HA, seg_pos={0 0}, upload=473901]: Uploading piece...
Dec 28 03:51:27.596421126 (540h22m27.467504161s) [noise=true , address=45.91.116.12:4225 , piece_id=524LIRRHRKYNBD6KZ5UMY4PRRA6UYHECHXWKUDKRI5AWZKSMCDDA, seg_pos={0 0}, upload=473901]: Uploading piece...
Dec 28 03:51:27.598406236 (540h22m27.469489271s) [noise=true , address=145.79.227.6:4258 , piece_id=TAPYNFVM4WFWDZXPVDQRU6EOQAUTXTBPHU4XXLCX33BDVUSRWBHQ, seg_pos={0 0}, upload=473901]: Uploading piece...
Dec 28 03:51:27.600303168 (540h22m27.471386213s) [noise=true , address=145.79.227.17:4971 , piece_id=3XMJTRCTFVNI43A4UEDZI6WQWSUJ4E4G22QUWBVNOUVDSNHPYCHA, seg_pos={0 0}, upload=473901]: Uploading piece...
Dec 28 03:51:27.602162148 (540h22m27.473245193s) [noise=true , address=145.79.227.2:4114 , piece_id=VAZZWHWQXOHMHBHUMEWYJSUXDNA5LHOCJJUZID2Q53CYU3DZDFAA, seg_pos={0 0}, upload=473901]: Uploading piece...
Dec 28 03:51:27.603346353 (540h22m27.474429388s) [noise=true , address=145.79.227.41:4086 , piece_id=IW3Q3HYE7QHN2YB3EHLI2VA7WCQPPREJVA3V3T6MLVX2FKWPIIYQ, seg_pos={0 0}, upload=473901]: Uploading piece...
Dec 28 03:51:27.604458230 (540h22m27.475541275s) [noise=true , address=145.79.227.45:4231 , piece_id=HW4XZI2DTIE5GPPYGGCWMDCTATOY5FIB6UKCO5QNMUEWT7UY6DNQ, seg_pos={0 0}, upload=473901]: Uploading piece...
Dec 28 03:51:27.605320803 (540h22m27.476403848s) [noise=true , address=77.90.31.18:4805 , piece_id=ZOVOQFVLT57IV2C4X22HZZPW2P2Q5ELPJMPUJZAOYIOLMMIAOHOA, seg_pos={0 0}, upload=473901]: Uploading piece...
Dec 28 03:51:29.311265605 (540h22m29.18234865s) [upload=473901]: Next returned nil segment
Dec 28 03:51:29.311295851 (540h22m29.182378896s) [upload=473901]: Waiting for error group managing segments...
Dec 28 03:51:29.592038785 (540h22m29.46312183s) [noise=true , address=145.79.227.3:4150 , piece_id=FIFEQJOEILOERQQUANU277W3CLXXSMYWIQGGZ6EUK4EMOSCDDB7Q, seg_pos={0 0}, upload=473901]: Done uploading piece. err: <nil>
Dec 28 03:51:29.600173434 (540h22m29.471256479s) [noise=true , address=145.79.227.22:4511 , piece_id=DOMEZRCPJLN2FPXWCNA2FSAAXSTNKKJJHWHKXLLT7AZWSSYPJSRQ, seg_pos={0 0}, upload=473901]: Done uploading piece. err: <nil>
Dec 28 03:51:29.609566420 (540h22m29.480649455s) [noise=true , address=145.79.227.45:4231 , piece_id=HW4XZI2DTIE5GPPYGGCWMDCTATOY5FIB6UKCO5QNMUEWT7UY6DNQ, seg_pos={0 0}, upload=473901]: Done uploading piece. err: <nil>
Dec 28 03:51:29.629261593 (540h22m29.500344638s) [noise=true , address=145.79.227.17:4971 , piece_id=3XMJTRCTFVNI43A4UEDZI6WQWSUJ4E4G22QUWBVNOUVDSNHPYCHA, seg_pos={0 0}, upload=473901]: Done uploading piece. err: <nil>
Dec 28 03:51:29.631478274 (540h22m29.502561309s) [noise=true , address=145.79.227.18:4006 , piece_id=WTBMCGGZDWOSGJ6GY54BV6L3CZTGL7SZMDBNFIMRCKTPSAUJRGVQ, seg_pos={0 0}, upload=473901]: Done uploading piece. err: <nil>
Dec 28 03:51:29.632762740 (540h22m29.503845775s) [noise=true , address=145.79.227.41:4086 , piece_id=IW3Q3HYE7QHN2YB3EHLI2VA7WCQPPREJVA3V3T6MLVX2FKWPIIYQ, seg_pos={0 0}, upload=473901]: Done uploading piece. err: <nil>
Dec 28 03:51:29.639142658 (540h22m29.510225693s) [noise=true , address=145.79.227.51:4410 , piece_id=SP6N5RLRL7EW6UTOBF74I2VROC3VEPJN6NK5KNCSZE6OHLHKN6HA, seg_pos={0 0}, upload=473901]: Done uploading piece. err: <nil>
Dec 28 03:51:29.639532701 (540h22m29.510615746s) [noise=true , address=145.79.227.50:4374 , piece_id=AP2UXSDPJEKO6YHNT46GKQGYWYX3MRB6L7I6X7YCANWQBROZALGA, seg_pos={0 0}, upload=473901]: Done uploading piece. err: <nil>
Dec 28 03:51:29.639741258 (540h22m29.510824293s) [noise=true , address=145.79.227.6:4258 , piece_id=TAPYNFVM4WFWDZXPVDQRU6EOQAUTXTBPHU4XXLCX33BDVUSRWBHQ, seg_pos={0 0}, upload=473901]: Done uploading piece. err: <nil>
Dec 28 03:51:29.891091092 (540h22m29.762174127s) [noise=true , address=45.91.116.4:4937 , piece_id=HU3AX6MJ4PEL3XZXYN2YED7RHZTCWIN2CCUUQGY4DC5ENIZDQS4A, seg_pos={0 0}, upload=473901]: Done uploading piece. err: <nil>
Dec 28 03:51:29.963943425 (540h22m29.83502647s) [noise=true , address=45.91.116.8:4045 , piece_id=QTRX4VGVPE6CWXSUO2UW22HWABMV72JLUM7TGBJTVPRALPQCSC6A, seg_pos={0 0}, upload=473901]: Done uploading piece. err: <nil>
Dec 28 03:51:29.976199793 (540h22m29.847282838s) [noise=true , address=45.91.116.14:4191 , piece_id=HT2V76RPE45TU4O5CB3VMCWQWYSUJKBG3LSNH2WWHI6HTD3D423A, seg_pos={0 0}, upload=473901]: Done uploading piece. err: <nil>
Dec 28 03:51:30.094966615 (540h22m29.966049651s) [noise=true , address=77.90.31.18:4805 , piece_id=ZOVOQFVLT57IV2C4X22HZZPW2P2Q5ELPJMPUJZAOYIOLMMIAOHOA, seg_pos={0 0}, upload=473901]: Done uploading piece. err: <nil>
Dec 28 03:51:30.094990150 (540h22m29.966073195s) [seg_pos={0 0}, upload=473901]: Segment reached optimal threshold of 13 pieces.
Dec 28 03:51:30.095170925 (540h22m29.96625396s) [noise=true , address=45.91.116.12:4225 , piece_id=524LIRRHRKYNBD6KZ5UMY4PRRA6UYHECHXWKUDKRI5AWZKSMCDDA, seg_pos={0 0}, upload=473901]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF
Dec 28 03:51:30.095181545 (540h22m29.96626458s) [noise=true , address=45.91.116.12:4225 , piece_id=524LIRRHRKYNBD6KZ5UMY4PRRA6UYHECHXWKUDKRI5AWZKSMCDDA, seg_pos={0 0}, upload=473901]: Long tail cancellation due to optimal threshold
Dec 28 03:51:30.095578129 (540h22m29.966661174s) [noise=true , address=145.79.227.2:4114 , piece_id=VAZZWHWQXOHMHBHUMEWYJSUXDNA5LHOCJJUZID2Q53CYU3DZDFAA, seg_pos={0 0}, upload=473901]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN): protocol: write tcp 188.166.225.170:38054->145.79.227.2:4114: use of closed network connection; write tcp 188.166.225.170:38054->145.79.227.2:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38054->145.79.227.2:4114: use of closed network connection
Dec 28 03:51:30.095587778 (540h22m29.966670813s) [noise=true , address=145.79.227.2:4114 , piece_id=VAZZWHWQXOHMHBHUMEWYJSUXDNA5LHOCJJUZID2Q53CYU3DZDFAA, seg_pos={0 0}, upload=473901]: Long tail cancellation due to optimal threshold
Dec 28 03:51:30.095933747 (540h22m29.967016782s) [noise=true , address=77.90.31.21:4880 , piece_id=6PHUE2QSDNDT3OR4QZLTUJAOCZSU4WWW64WMM74YAOYVTQGIMEQA, seg_pos={0 0}, upload=473901]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: read tcp 188.166.225.170:45884->77.90.31.21:4880: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 28 03:51:30.095941061 (540h22m29.967024096s) [noise=true , address=77.90.31.21:4880 , piece_id=6PHUE2QSDNDT3OR4QZLTUJAOCZSU4WWW64WMM74YAOYVTQGIMEQA, seg_pos={0 0}, upload=473901]: Long tail cancellation due to optimal threshold
Dec 28 03:51:30.095983922 (540h22m29.967066957s) [seg_pos={0 0}, upload=473901]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 28 03:51:30.096074856 (540h22m29.967157891s) [upload=473901]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 03:51:30.096079815 (540h22m29.96716285s) [upload=473901]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 03:51:30.392898018 (540h22m30.263981053s) [upload=473901]: Done uploading segments...
Dec 28 03:51:30.392924478 (540h22m30.264007523s) [upload=473901]: Upload finished. err: <nil>
Duration: 30.00331524s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
20 out of 60 is failed (threshold=6) [....X.....XX.....X..X.X....X.XXXX....X.X.X...X..X.X.X..X...X] duration:
5.001654396s trace:
https://jaeger.dev.storj.io/trace/3f5f67286c8d6dcc
FAIL (2025-12-28 03:51:51 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.00168307s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.............................................X..............] duration:
16.136518996s trace:
https://jaeger.dev.storj.io/trace/3b8130f32e60318d
FAIL (2025-12-28 03:51:36 UTC)
orbiter warning: context deadline exceeded
Duration: 16.136553773s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
50 out of 60 is failed (threshold=3) [.X..XX..XXXXXX.XXXXXXXXXXXXXXXXX.XXXXX.XXXXXXXXXXXXXXXXX..XX] duration:
30.002668225s trace:
https://jaeger.dev.storj.io/trace/743f339e2c583ee3
FAIL (2025-12-28 03:51:26 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: Y7N4U6ULZTZRLSCUO4GBD7PTYM6F27YVMLZ7MB3QNQU2GARQ7DLQ) context deadline exceeded; piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: A3FRAWNDEL6RPFIJO2DZ6SSJHDH6BC3ZV2RXZ55SY73EXRFYGAIA) context deadline exceeded
======LOGS======
8 03:50:57.156202030 (540h21m57.027285076s) [upload=473893]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 28 03:50:57.427754532 (540h21m57.298837568s) [seg_pos={0 0}, upload=473893]: Begin upload segment...
Dec 28 03:50:57.427820889 (540h21m57.298903924s) [seg_pos={0 0}, upload=473893]: Done begin upload segment.
Dec 28 03:50:57.427866826 (540h21m57.298949861s) [noise=true , address=145.79.227.13:4804 , piece_id=WP3DTTFIAG33VP57IS3RMWPJ5PWD6YJVSGNHEISJXMDK7TQGUYSQ, seg_pos={0 0}, upload=473893]: Uploading piece...
Dec 28 03:50:57.427909939 (540h21m57.298992974s) [noise=true , address=145.79.227.3:4151 , piece_id=7RZU2RPCYIL7EWIVHVGVDCJHH7VDFOALPBFNP6OTDT4IUPJFK5WA, seg_pos={0 0}, upload=473893]: Uploading piece...
Dec 28 03:50:57.428438044 (540h21m57.299521089s) [noise=true , address=145.79.227.31:4798 , piece_id=KE4Q2XYOARORNNCHJAECMYFAVXVWVTLKHSL7EZDFZ3OASZCTY6XQ, seg_pos={0 0}, upload=473893]: Uploading piece...
Dec 28 03:50:57.428746782 (540h21m57.299829827s) [noise=true , address=145.79.227.41:4087 , piece_id=LFBEABCHHXNDOCKHUH2S4GWEJCUW25ACJ2UHTJYAFMBTPJWM4AEQ, seg_pos={0 0}, upload=473893]: Uploading piece...
Dec 28 03:50:57.429067413 (540h21m57.300150448s) [noise=true , address=45.91.116.6:4976 , piece_id=G3YHUMXR5JVGMQGGF55N6BLZ37WAZSQWJWYLULQOP3YJX7O3VBBQ, seg_pos={0 0}, upload=473893]: Uploading piece...
Dec 28 03:50:57.429375419 (540h21m57.300458454s) [noise=true , address=45.91.116.9:4264 , piece_id=UX5L742PDOWZRB7LFYN7BYICYGYQDN35PB762GGPSYS2QYWCM7NQ, seg_pos={0 0}, upload=473893]: Uploading piece...
Dec 28 03:50:57.429641345 (540h21m57.30072438s) [noise=true , address=145.79.227.45:4229 , piece_id=NO4RBYNFEHAX7E7E6C7SOAKYH4HSNHHBCFWVKUSORWQ3CI4LBQ5A, seg_pos={0 0}, upload=473893]: Uploading piece...
Dec 28 03:50:57.429974029 (540h21m57.301057064s) [noise=true , address=145.79.227.9:4365 , piece_id=WCAWNK4W3652LPIN3V74WZOJQNARYIUC2D7EG7O35WTBJ3Q6EFSA, seg_pos={0 0}, upload=473893]: Uploading piece...
Dec 28 03:50:57.430358150 (540h21m57.301441185s) [noise=true , address=45.91.116.8:4046 , piece_id=Y7N4U6ULZTZRLSCUO4GBD7PTYM6F27YVMLZ7MB3QNQU2GARQ7DLQ, seg_pos={0 0}, upload=473893]: Uploading piece...
Dec 28 03:50:57.431844521 (540h21m57.302927556s) [noise=true , address=45.91.116.14:4192 , piece_id=L76AIG5UPUZ5HMARI4UUS66MW6NVTUYTJRO4MLZPAY7MWWPLJWGQ, seg_pos={0 0}, upload=473893]: Uploading piece...
Dec 28 03:50:57.432949304 (540h21m57.304032339s) [noise=true , address=145.79.227.51:4411 , piece_id=CZPDGXKUHBFPXYVBHWKRWYTQOTQIMLITH2R7S6DUH3P4FZS7YGWA, seg_pos={0 0}, upload=473893]: Uploading piece...
Dec 28 03:50:57.434683667 (540h21m57.305766702s) [noise=true , address=45.91.116.12:4227 , piece_id=A3FRAWNDEL6RPFIJO2DZ6SSJHDH6BC3ZV2RXZ55SY73EXRFYGAIA, seg_pos={0 0}, upload=473893]: Uploading piece...
Dec 28 03:50:57.436233929 (540h21m57.307316964s) [noise=true , address=45.91.116.11:4120 , piece_id=M4FLURND6XHEJQA4SWGOTNYQR7V2Y6TF4W5NQUZOJXKTJB5IFQVA, seg_pos={0 0}, upload=473893]: Uploading piece...
Dec 28 03:50:57.439576063 (540h21m57.310659098s) [noise=true , address=45.91.116.5:4298 , piece_id=XRV2U5DYQGH2APWVX37TSAC4B2OWQMSJWG7PL4LCZKDL6JQQGAIA, seg_pos={0 0}, upload=473893]: Uploading piece...
Dec 28 03:50:57.440711394 (540h21m57.311794439s) [noise=true , address=145.79.227.38:4977 , piece_id=I3BTQYYC7HIYQZ6PCMMCKKSBX5GNNV7HA3AM5N2W5JUNCT2VGZ7A, seg_pos={0 0}, upload=473893]: Uploading piece...
Dec 28 03:50:57.442843305 (540h21m57.31392637s) [noise=true , address=145.79.227.7:4295 , piece_id=SLLY5PKU72UQGJIBDOPCUT26XCSQXJD6UXATIAZQEYTLLOU4IFWQ, seg_pos={0 0}, upload=473893]: Uploading piece...
Dec 28 03:50:59.162243777 (540h21m59.033326812s) [upload=473893]: Next returned nil segment
Dec 28 03:50:59.162255450 (540h21m59.033338485s) [upload=473893]: Waiting for error group managing segments...
Dec 28 03:50:59.438866925 (540h21m59.30994996s) [noise=true , address=145.79.227.13:4804 , piece_id=WP3DTTFIAG33VP57IS3RMWPJ5PWD6YJVSGNHEISJXMDK7TQGUYSQ, seg_pos={0 0}, upload=473893]: Done uploading piece. err: <nil>
Dec 28 03:50:59.450988087 (540h21m59.322071122s) [noise=true , address=145.79.227.9:4365 , piece_id=WCAWNK4W3652LPIN3V74WZOJQNARYIUC2D7EG7O35WTBJ3Q6EFSA, seg_pos={0 0}, upload=473893]: Done uploading piece. err: <nil>
Dec 28 03:50:59.455410778 (540h21m59.326493813s) [noise=true , address=145.79.227.31:4798 , piece_id=KE4Q2XYOARORNNCHJAECMYFAVXVWVTLKHSL7EZDFZ3OASZCTY6XQ, seg_pos={0 0}, upload=473893]: Done uploading piece. err: <nil>
Dec 28 03:50:59.463473580 (540h21m59.334556615s) [noise=true , address=145.79.227.45:4229 , piece_id=NO4RBYNFEHAX7E7E6C7SOAKYH4HSNHHBCFWVKUSORWQ3CI4LBQ5A, seg_pos={0 0}, upload=473893]: Done uploading piece. err: <nil>
Dec 28 03:50:59.469286209 (540h21m59.340369244s) [noise=true , address=145.79.227.41:4087 , piece_id=LFBEABCHHXNDOCKHUH2S4GWEJCUW25ACJ2UHTJYAFMBTPJWM4AEQ, seg_pos={0 0}, upload=473893]: Done uploading piece. err: <nil>
Dec 28 03:50:59.476754019 (540h21m59.347837064s) [noise=true , address=145.79.227.38:4977 , piece_id=I3BTQYYC7HIYQZ6PCMMCKKSBX5GNNV7HA3AM5N2W5JUNCT2VGZ7A, seg_pos={0 0}, upload=473893]: Done uploading piece. err: <nil>
Dec 28 03:50:59.566245726 (540h21m59.437328761s) [noise=true , address=145.79.227.7:4295 , piece_id=SLLY5PKU72UQGJIBDOPCUT26XCSQXJD6UXATIAZQEYTLLOU4IFWQ, seg_pos={0 0}, upload=473893]: Done uploading piece. err: <nil>
Dec 28 03:50:59.670250860 (540h21m59.541333895s) [noise=true , address=145.79.227.3:4151 , piece_id=7RZU2RPCYIL7EWIVHVGVDCJHH7VDFOALPBFNP6OTDT4IUPJFK5WA, seg_pos={0 0}, upload=473893]: Done uploading piece. err: <nil>
Dec 28 03:50:59.711961584 (540h21m59.583044629s) [noise=true , address=45.91.116.12:4227 , piece_id=A3FRAWNDEL6RPFIJO2DZ6SSJHDH6BC3ZV2RXZ55SY73EXRFYGAIA, seg_pos={0 0}, upload=473893]: Done uploading piece. err: <nil>
Dec 28 03:50:59.744567933 (540h21m59.615650968s) [noise=true , address=45.91.116.6:4976 , piece_id=G3YHUMXR5JVGMQGGF55N6BLZ37WAZSQWJWYLULQOP3YJX7O3VBBQ, seg_pos={0 0}, upload=473893]: Done uploading piece. err: <nil>
Dec 28 03:50:59.795080567 (540h21m59.666163662s) [noise=true , address=45.91.116.14:4192 , piece_id=L76AIG5UPUZ5HMARI4UUS66MW6NVTUYTJRO4MLZPAY7MWWPLJWGQ, seg_pos={0 0}, upload=473893]: Done uploading piece. err: <nil>
Dec 28 03:51:00.392357876 (540h22m0.263440921s) [noise=true , address=45.91.116.11:4120 , piece_id=M4FLURND6XHEJQA4SWGOTNYQR7V2Y6TF4W5NQUZOJXKTJB5IFQVA, seg_pos={0 0}, upload=473893]: Done uploading piece. err: <nil>
Dec 28 03:51:00.422085562 (540h22m0.293168597s) [noise=true , address=45.91.116.8:4046 , piece_id=Y7N4U6ULZTZRLSCUO4GBD7PTYM6F27YVMLZ7MB3QNQU2GARQ7DLQ, seg_pos={0 0}, upload=473893]: Done uploading piece. err: <nil>
Dec 28 03:51:00.422127121 (540h22m0.293210166s) [seg_pos={0 0}, upload=473893]: Segment reached optimal threshold of 13 pieces.
Dec 28 03:51:00.422334526 (540h22m0.293417561s) [noise=true , address=45.91.116.5:4298 , piece_id=XRV2U5DYQGH2APWVX37TSAC4B2OWQMSJWG7PL4LCZKDL6JQQGAIA, seg_pos={0 0}, upload=473893]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: write tcp 188.166.225.170:59378->45.91.116.5:4298: use of closed network connection; write tcp 188.166.225.170:59378->45.91.116.5:4298: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59378->45.91.116.5:4298: use of closed network connection
Dec 28 03:51:00.422343844 (540h22m0.293426879s) [noise=true , address=45.91.116.5:4298 , piece_id=XRV2U5DYQGH2APWVX37TSAC4B2OWQMSJWG7PL4LCZKDL6JQQGAIA, seg_pos={0 0}, upload=473893]: Long tail cancellation due to optimal threshold
Dec 28 03:51:00.422420280 (540h22m0.293503315s) [noise=true , address=45.91.116.9:4264 , piece_id=UX5L742PDOWZRB7LFYN7BYICYGYQDN35PB762GGPSYS2QYWCM7NQ, seg_pos={0 0}, upload=473893]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: write tcp 188.166.225.170:56154->45.91.116.9:4264: use of closed network connection; write tcp 188.166.225.170:56154->45.91.116.9:4264: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56154->45.91.116.9:4264: use of closed network connection
Dec 28 03:51:00.422427864 (540h22m0.293510899s) [noise=true , address=45.91.116.9:4264 , piece_id=UX5L742PDOWZRB7LFYN7BYICYGYQDN35PB762GGPSYS2QYWCM7NQ, seg_pos={0 0}, upload=473893]: Long tail cancellation due to optimal threshold
Dec 28 03:51:00.422478250 (540h22m0.293561295s) [noise=true , address=145.79.227.51:4411 , piece_id=CZPDGXKUHBFPXYVBHWKRWYTQOTQIMLITH2R7S6DUH3P4FZS7YGWA, seg_pos={0 0}, upload=473893]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DT68m6J4PpDk3XUDhDaaqV46NLKVz4A23716JLFxEAh122GeR): protocol: write tcp 188.166.225.170:42900->145.79.227.51:4411: use of closed network connection; write tcp 188.166.225.170:42900->145.79.227.51:4411: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42900->145.79.227.51:4411: use of closed network connection
Dec 28 03:51:00.422486575 (540h22m0.29356961s) [noise=true , address=145.79.227.51:4411 , piece_id=CZPDGXKUHBFPXYVBHWKRWYTQOTQIMLITH2R7S6DUH3P4FZS7YGWA, seg_pos={0 0}, upload=473893]: Long tail cancellation due to optimal threshold
Dec 28 03:51:00.422731432 (540h22m0.293814477s) [seg_pos={0 0}, upload=473893]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 28 03:51:00.422829669 (540h22m0.293912704s) [upload=473893]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 03:51:00.422837675 (540h22m0.29392071s) [upload=473893]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 03:51:00.822736861 (540h22m0.693819906s) [upload=473893]: Done uploading segments...
Dec 28 03:51:00.822762659 (540h22m0.693845695s) [upload=473893]: Upload finished. err: <nil>
Duration: 30.002789577s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [......................X...X.................................] duration:
8.615743231s trace:
https://jaeger.dev.storj.io/trace/54422f75ff432c0
FAIL (2025-12-28 03:51:23 UTC)
orbiter warning:
operation: DeleteObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
, piece_id=SPQSUVHIHI32ZMIRE4NHSNJCMWS4NS6ZZPYF6FV7FQH6VADQIV5A, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:35.720215866 (539h54m35.591298911s) [noise=true , address=145.79.227.43:4187 , piece_id=MRILXYFSWVSH35ZCCFZWLEXENWJII6G6LOR2N277GIBMI4PXPPTQ, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:35.720567665 (539h54m35.5916507s) [noise=true , address=145.79.227.33:4896 , piece_id=7NRM7I4IDHW2WGOCG3B6L6HN74ANXWDJCFRPBM4BCHWJELSNNUXA, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:35.720883607 (539h54m35.591966652s) [noise=true , address=145.79.227.40:4062 , piece_id=RCVKQLDSWIWX2FQBTUFPEUXQZLP2C3ZCLV2JWHOV7CMJ7ZBNWZNQ, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:35.721874904 (539h54m35.592957949s) [noise=true , address=64.49.13.140:4513 , piece_id=Y3HW5I2FX2CWSPVUS7CU3JPNN2RHP2ASL757AUYINBKRKCLXAJGQ, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:35.722249658 (539h54m35.593332693s) [noise=true , address=64.49.13.169:4715 , piece_id=FRVNWFP2AE4DXE4RKZRE6WTK43T74BTTYM2U7E4P3B4XZIMC2QMQ, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:35.722786881 (539h54m35.593869926s) [noise=true , address=77.90.31.14:4673 , piece_id=ZIB5VOFL7PJSRFDV3HP5XAKYHOJXRVHCYDPIHKRYPNM5ROAY5DTQ, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:35.723112100 (539h54m35.594195135s) [noise=true , address=77.90.31.17:4779 , piece_id=TLU5ZLZRVAMGQLSE2LLNIUE4WOWJBMFJGFQKC3KOUJ23V3UVFMRA, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:35.723440756 (539h54m35.594523791s) [noise=true , address=145.79.227.37:4569 , piece_id=DUXSQUT36XUF66ZVQ2V7ZWNXA6JZKBDO5RXT3QVT26GE2FTH3QXQ, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:35.723849544 (539h54m35.594932589s) [noise=true , address=145.79.227.27:4723 , piece_id=4DECSGDNZLXNQVPU7QALLPFQYDO7MOG6TIBNWQ5UHNVCIHB4GUYQ, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:35.724241290 (539h54m35.595324335s) [noise=true , address=77.90.31.11:4587 , piece_id=6467VZCLWHJTZ36ENMEOEOU2E4MAUCDL2XHNAOMJTCCTUSN33NFA, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:35.724607808 (539h54m35.595690854s) [noise=true , address=145.79.227.47:4272 , piece_id=FL2U4W2YCV5ANDVFHB7H4TFT3SYTBGNLI3KZDI7MY2K6NTSMRUAA, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:35.724974035 (539h54m35.59605707s) [noise=true , address=145.79.227.18:4040 , piece_id=W55GK3QDHRHNR6YJ7JKVZMIEKFQLZSTXKSQ5OMW3U7XRI44EM6RA, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:35.725302281 (539h54m35.596385316s) [noise=true , address=145.79.227.38:4989 , piece_id=OUINZ6BXOLFCDNCXMX7PPFSKWMJ7IPZWO7DAGLGVV2AB7UHO3JYQ, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:35.726824420 (539h54m35.597907455s) [noise=true , address=145.79.227.26:4669 , piece_id=5D7SGVO274FEZ4JVESTMUYQ3RZ3DJA5DL2QZV5YKMXEB4TM6PHAA, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:37.063163473 (539h54m36.934246518s) [upload=473490]: Next returned nil segment
Dec 28 03:23:37.063176499 (539h54m36.934259534s) [upload=473490]: Waiting for error group managing segments...
Dec 28 03:23:37.416516311 (539h54m37.287599376s) [noise=true , address=64.49.13.140:4513 , piece_id=Y3HW5I2FX2CWSPVUS7CU3JPNN2RHP2ASL757AUYINBKRKCLXAJGQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil>
Dec 28 03:23:37.606629586 (539h54m37.477712622s) [noise=true , address=145.79.227.27:4723 , piece_id=4DECSGDNZLXNQVPU7QALLPFQYDO7MOG6TIBNWQ5UHNVCIHB4GUYQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil>
Dec 28 03:23:37.617730204 (539h54m37.488813249s) [noise=true , address=145.79.227.26:4669 , piece_id=5D7SGVO274FEZ4JVESTMUYQ3RZ3DJA5DL2QZV5YKMXEB4TM6PHAA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil>
Dec 28 03:23:37.632542950 (539h54m37.503625995s) [noise=true , address=145.79.227.38:4989 , piece_id=OUINZ6BXOLFCDNCXMX7PPFSKWMJ7IPZWO7DAGLGVV2AB7UHO3JYQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil>
Dec 28 03:23:37.643987531 (539h54m37.515070566s) [noise=true , address=145.79.227.47:4272 , piece_id=FL2U4W2YCV5ANDVFHB7H4TFT3SYTBGNLI3KZDI7MY2K6NTSMRUAA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil>
Dec 28 03:23:37.652913619 (539h54m37.523996654s) [noise=true , address=145.79.227.18:4040 , piece_id=W55GK3QDHRHNR6YJ7JKVZMIEKFQLZSTXKSQ5OMW3U7XRI44EM6RA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil>
Dec 28 03:23:37.661966357 (539h54m37.533049392s) [noise=true , address=145.79.227.37:4569 , piece_id=DUXSQUT36XUF66ZVQ2V7ZWNXA6JZKBDO5RXT3QVT26GE2FTH3QXQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil>
Dec 28 03:23:37.778627848 (539h54m37.649710893s) [noise=true , address=77.90.31.11:4561 , piece_id=HIUCZRRZOMU4RBYZU5SADJAOS4RVML2I6PILIX2V65RS22EBN42Q, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil>
Dec 28 03:23:37.815974812 (539h54m37.687057857s) [noise=true , address=45.91.116.14:4217 , piece_id=SPQSUVHIHI32ZMIRE4NHSNJCMWS4NS6ZZPYF6FV7FQH6VADQIV5A, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil>
Dec 28 03:23:37.861731780 (539h54m37.732814815s) [noise=true , address=77.90.31.14:4673 , piece_id=ZIB5VOFL7PJSRFDV3HP5XAKYHOJXRVHCYDPIHKRYPNM5ROAY5DTQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil>
Dec 28 03:23:38.093273211 (539h54m37.964356256s) [noise=true , address=77.90.31.11:4587 , piece_id=6467VZCLWHJTZ36ENMEOEOU2E4MAUCDL2XHNAOMJTCCTUSN33NFA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil>
Dec 28 03:23:38.267694395 (539h54m38.13877745s) [noise=true , address=77.90.31.17:4779 , piece_id=TLU5ZLZRVAMGQLSE2LLNIUE4WOWJBMFJGFQKC3KOUJ23V3UVFMRA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil>
Dec 28 03:23:38.540947761 (539h54m38.412030796s) [noise=true , address=145.79.227.40:4062 , piece_id=RCVKQLDSWIWX2FQBTUFPEUXQZLP2C3ZCLV2JWHOV7CMJ7ZBNWZNQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil>
Dec 28 03:23:38.815393527 (539h54m38.686476562s) [noise=true , address=145.79.227.43:4187 , piece_id=MRILXYFSWVSH35ZCCFZWLEXENWJII6G6LOR2N277GIBMI4PXPPTQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil>
Dec 28 03:23:38.815412894 (539h54m38.686495939s) [seg_pos={0 0}, upload=473490]: Segment reached optimal threshold of 14 pieces.
Dec 28 03:23:38.815594951 (539h54m38.686677986s) [noise=true , address=64.49.13.169:4715 , piece_id=FRVNWFP2AE4DXE4RKZRE6WTK43T74BTTYM2U7E4P3B4XZIMC2QMQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:43502->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:43502->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43502->64.49.13.169:4715: use of closed network connection
Dec 28 03:23:38.815601032 (539h54m38.686684077s) [noise=true , address=64.49.13.169:4715 , piece_id=FRVNWFP2AE4DXE4RKZRE6WTK43T74BTTYM2U7E4P3B4XZIMC2QMQ, seg_pos={0 0}, upload=473490]: Long tail cancellation due to optimal threshold
Dec 28 03:23:38.815826983 (539h54m38.686910018s) [noise=true , address=145.79.227.41:4118 , piece_id=C7JJ3NHR5355M64G2EF2WWERQVGWR243RQHDV3GBAIC3A66JDMHQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128w37UHxJB9R49iwmHbATt3bXUb1FMTQJ3Wjka7So7eqKWg4iF): protocol: expected piece hash; context canceled; EOF
Dec 28 03:23:38.815834507 (539h54m38.686917542s) [noise=true , address=145.79.227.41:4118 , piece_id=C7JJ3NHR5355M64G2EF2WWERQVGWR243RQHDV3GBAIC3A66JDMHQ, seg_pos={0 0}, upload=473490]: Long tail cancellation due to optimal threshold
Dec 28 03:23:38.815890433 (539h54m38.686973468s) [noise=true , address=145.79.227.33:4896 , piece_id=7NRM7I4IDHW2WGOCG3B6L6HN74ANXWDJCFRPBM4BCHWJELSNNUXA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ndSgjgfZMVQni5tw1ugdTrhJt4c4NagdvgYTbSbjatYYtfu1o): protocol: expected piece hash; context canceled; EOF
Dec 28 03:23:38.815914619 (539h54m38.686997654s) [noise=true , address=145.79.227.33:4896 , piece_id=7NRM7I4IDHW2WGOCG3B6L6HN74ANXWDJCFRPBM4BCHWJELSNNUXA, seg_pos={0 0}, upload=473490]: Long tail cancellation due to optimal threshold
Dec 28 03:23:38.816105613 (539h54m38.687188648s) [noise=true , address=145.79.227.28:4731 , piece_id=EPW3LUA3EWRCURHZ66NCRCA36CNIW4GYQMFNKPZZTRCB3YBGTKJA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NQzTwVYQa6ZzxAHQm1apkwWnqJstTnAJQyHKhBzC2jTN9kEVK): protocol: write tcp 188.166.225.170:58940->145.79.227.28:4731: use of closed network connection; write tcp 188.166.225.170:58940->145.79.227.28:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58940->145.79.227.28:4731: use of closed network connection
Dec 28 03:23:38.816112426 (539h54m38.687195461s) [noise=true , address=145.79.227.28:4731 , piece_id=EPW3LUA3EWRCURHZ66NCRCA36CNIW4GYQMFNKPZZTRCB3YBGTKJA, seg_pos={0 0}, upload=473490]: Long tail cancellation due to optimal threshold
Dec 28 03:23:38.816128436 (539h54m38.687211481s) [seg_pos={0 0}, upload=473490]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 28 03:23:38.816257582 (539h54m38.687340627s) [upload=473490]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 03:23:38.816265778 (539h54m38.687348813s) [upload=473490]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 03:23:39.111115407 (539h54m38.982198442s) [upload=473490]: Done uploading segments...
Dec 28 03:23:39.111138972 (539h54m38.982222008s) [upload=473490]: Upload finished. err: <nil>
Duration: 8.615764212s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [.......................X...X................................] duration:
56.724244167s trace:
https://jaeger.dev.storj.io/trace/54422f75ff432c0
FAIL (2025-12-28 03:51:14 UTC)
orbiter warning:
operation: DeleteObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
, piece_id=SPQSUVHIHI32ZMIRE4NHSNJCMWS4NS6ZZPYF6FV7FQH6VADQIV5A, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:35.720215866 (539h54m35.591298911s) [noise=true , address=145.79.227.43:4187 , piece_id=MRILXYFSWVSH35ZCCFZWLEXENWJII6G6LOR2N277GIBMI4PXPPTQ, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:35.720567665 (539h54m35.5916507s) [noise=true , address=145.79.227.33:4896 , piece_id=7NRM7I4IDHW2WGOCG3B6L6HN74ANXWDJCFRPBM4BCHWJELSNNUXA, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:35.720883607 (539h54m35.591966652s) [noise=true , address=145.79.227.40:4062 , piece_id=RCVKQLDSWIWX2FQBTUFPEUXQZLP2C3ZCLV2JWHOV7CMJ7ZBNWZNQ, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:35.721874904 (539h54m35.592957949s) [noise=true , address=64.49.13.140:4513 , piece_id=Y3HW5I2FX2CWSPVUS7CU3JPNN2RHP2ASL757AUYINBKRKCLXAJGQ, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:35.722249658 (539h54m35.593332693s) [noise=true , address=64.49.13.169:4715 , piece_id=FRVNWFP2AE4DXE4RKZRE6WTK43T74BTTYM2U7E4P3B4XZIMC2QMQ, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:35.722786881 (539h54m35.593869926s) [noise=true , address=77.90.31.14:4673 , piece_id=ZIB5VOFL7PJSRFDV3HP5XAKYHOJXRVHCYDPIHKRYPNM5ROAY5DTQ, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:35.723112100 (539h54m35.594195135s) [noise=true , address=77.90.31.17:4779 , piece_id=TLU5ZLZRVAMGQLSE2LLNIUE4WOWJBMFJGFQKC3KOUJ23V3UVFMRA, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:35.723440756 (539h54m35.594523791s) [noise=true , address=145.79.227.37:4569 , piece_id=DUXSQUT36XUF66ZVQ2V7ZWNXA6JZKBDO5RXT3QVT26GE2FTH3QXQ, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:35.723849544 (539h54m35.594932589s) [noise=true , address=145.79.227.27:4723 , piece_id=4DECSGDNZLXNQVPU7QALLPFQYDO7MOG6TIBNWQ5UHNVCIHB4GUYQ, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:35.724241290 (539h54m35.595324335s) [noise=true , address=77.90.31.11:4587 , piece_id=6467VZCLWHJTZ36ENMEOEOU2E4MAUCDL2XHNAOMJTCCTUSN33NFA, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:35.724607808 (539h54m35.595690854s) [noise=true , address=145.79.227.47:4272 , piece_id=FL2U4W2YCV5ANDVFHB7H4TFT3SYTBGNLI3KZDI7MY2K6NTSMRUAA, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:35.724974035 (539h54m35.59605707s) [noise=true , address=145.79.227.18:4040 , piece_id=W55GK3QDHRHNR6YJ7JKVZMIEKFQLZSTXKSQ5OMW3U7XRI44EM6RA, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:35.725302281 (539h54m35.596385316s) [noise=true , address=145.79.227.38:4989 , piece_id=OUINZ6BXOLFCDNCXMX7PPFSKWMJ7IPZWO7DAGLGVV2AB7UHO3JYQ, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:35.726824420 (539h54m35.597907455s) [noise=true , address=145.79.227.26:4669 , piece_id=5D7SGVO274FEZ4JVESTMUYQ3RZ3DJA5DL2QZV5YKMXEB4TM6PHAA, seg_pos={0 0}, upload=473490]: Uploading piece...
Dec 28 03:23:37.063163473 (539h54m36.934246518s) [upload=473490]: Next returned nil segment
Dec 28 03:23:37.063176499 (539h54m36.934259534s) [upload=473490]: Waiting for error group managing segments...
Dec 28 03:23:37.416516311 (539h54m37.287599376s) [noise=true , address=64.49.13.140:4513 , piece_id=Y3HW5I2FX2CWSPVUS7CU3JPNN2RHP2ASL757AUYINBKRKCLXAJGQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil>
Dec 28 03:23:37.606629586 (539h54m37.477712622s) [noise=true , address=145.79.227.27:4723 , piece_id=4DECSGDNZLXNQVPU7QALLPFQYDO7MOG6TIBNWQ5UHNVCIHB4GUYQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil>
Dec 28 03:23:37.617730204 (539h54m37.488813249s) [noise=true , address=145.79.227.26:4669 , piece_id=5D7SGVO274FEZ4JVESTMUYQ3RZ3DJA5DL2QZV5YKMXEB4TM6PHAA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil>
Dec 28 03:23:37.632542950 (539h54m37.503625995s) [noise=true , address=145.79.227.38:4989 , piece_id=OUINZ6BXOLFCDNCXMX7PPFSKWMJ7IPZWO7DAGLGVV2AB7UHO3JYQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil>
Dec 28 03:23:37.643987531 (539h54m37.515070566s) [noise=true , address=145.79.227.47:4272 , piece_id=FL2U4W2YCV5ANDVFHB7H4TFT3SYTBGNLI3KZDI7MY2K6NTSMRUAA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil>
Dec 28 03:23:37.652913619 (539h54m37.523996654s) [noise=true , address=145.79.227.18:4040 , piece_id=W55GK3QDHRHNR6YJ7JKVZMIEKFQLZSTXKSQ5OMW3U7XRI44EM6RA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil>
Dec 28 03:23:37.661966357 (539h54m37.533049392s) [noise=true , address=145.79.227.37:4569 , piece_id=DUXSQUT36XUF66ZVQ2V7ZWNXA6JZKBDO5RXT3QVT26GE2FTH3QXQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil>
Dec 28 03:23:37.778627848 (539h54m37.649710893s) [noise=true , address=77.90.31.11:4561 , piece_id=HIUCZRRZOMU4RBYZU5SADJAOS4RVML2I6PILIX2V65RS22EBN42Q, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil>
Dec 28 03:23:37.815974812 (539h54m37.687057857s) [noise=true , address=45.91.116.14:4217 , piece_id=SPQSUVHIHI32ZMIRE4NHSNJCMWS4NS6ZZPYF6FV7FQH6VADQIV5A, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil>
Dec 28 03:23:37.861731780 (539h54m37.732814815s) [noise=true , address=77.90.31.14:4673 , piece_id=ZIB5VOFL7PJSRFDV3HP5XAKYHOJXRVHCYDPIHKRYPNM5ROAY5DTQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil>
Dec 28 03:23:38.093273211 (539h54m37.964356256s) [noise=true , address=77.90.31.11:4587 , piece_id=6467VZCLWHJTZ36ENMEOEOU2E4MAUCDL2XHNAOMJTCCTUSN33NFA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil>
Dec 28 03:23:38.267694395 (539h54m38.13877745s) [noise=true , address=77.90.31.17:4779 , piece_id=TLU5ZLZRVAMGQLSE2LLNIUE4WOWJBMFJGFQKC3KOUJ23V3UVFMRA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil>
Dec 28 03:23:38.540947761 (539h54m38.412030796s) [noise=true , address=145.79.227.40:4062 , piece_id=RCVKQLDSWIWX2FQBTUFPEUXQZLP2C3ZCLV2JWHOV7CMJ7ZBNWZNQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil>
Dec 28 03:23:38.815393527 (539h54m38.686476562s) [noise=true , address=145.79.227.43:4187 , piece_id=MRILXYFSWVSH35ZCCFZWLEXENWJII6G6LOR2N277GIBMI4PXPPTQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil>
Dec 28 03:23:38.815412894 (539h54m38.686495939s) [seg_pos={0 0}, upload=473490]: Segment reached optimal threshold of 14 pieces.
Dec 28 03:23:38.815594951 (539h54m38.686677986s) [noise=true , address=64.49.13.169:4715 , piece_id=FRVNWFP2AE4DXE4RKZRE6WTK43T74BTTYM2U7E4P3B4XZIMC2QMQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:43502->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:43502->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43502->64.49.13.169:4715: use of closed network connection
Dec 28 03:23:38.815601032 (539h54m38.686684077s) [noise=true , address=64.49.13.169:4715 , piece_id=FRVNWFP2AE4DXE4RKZRE6WTK43T74BTTYM2U7E4P3B4XZIMC2QMQ, seg_pos={0 0}, upload=473490]: Long tail cancellation due to optimal threshold
Dec 28 03:23:38.815826983 (539h54m38.686910018s) [noise=true , address=145.79.227.41:4118 , piece_id=C7JJ3NHR5355M64G2EF2WWERQVGWR243RQHDV3GBAIC3A66JDMHQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128w37UHxJB9R49iwmHbATt3bXUb1FMTQJ3Wjka7So7eqKWg4iF): protocol: expected piece hash; context canceled; EOF
Dec 28 03:23:38.815834507 (539h54m38.686917542s) [noise=true , address=145.79.227.41:4118 , piece_id=C7JJ3NHR5355M64G2EF2WWERQVGWR243RQHDV3GBAIC3A66JDMHQ, seg_pos={0 0}, upload=473490]: Long tail cancellation due to optimal threshold
Dec 28 03:23:38.815890433 (539h54m38.686973468s) [noise=true , address=145.79.227.33:4896 , piece_id=7NRM7I4IDHW2WGOCG3B6L6HN74ANXWDJCFRPBM4BCHWJELSNNUXA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ndSgjgfZMVQni5tw1ugdTrhJt4c4NagdvgYTbSbjatYYtfu1o): protocol: expected piece hash; context canceled; EOF
Dec 28 03:23:38.815914619 (539h54m38.686997654s) [noise=true , address=145.79.227.33:4896 , piece_id=7NRM7I4IDHW2WGOCG3B6L6HN74ANXWDJCFRPBM4BCHWJELSNNUXA, seg_pos={0 0}, upload=473490]: Long tail cancellation due to optimal threshold
Dec 28 03:23:38.816105613 (539h54m38.687188648s) [noise=true , address=145.79.227.28:4731 , piece_id=EPW3LUA3EWRCURHZ66NCRCA36CNIW4GYQMFNKPZZTRCB3YBGTKJA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NQzTwVYQa6ZzxAHQm1apkwWnqJstTnAJQyHKhBzC2jTN9kEVK): protocol: write tcp 188.166.225.170:58940->145.79.227.28:4731: use of closed network connection; write tcp 188.166.225.170:58940->145.79.227.28:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58940->145.79.227.28:4731: use of closed network connection
Dec 28 03:23:38.816112426 (539h54m38.687195461s) [noise=true , address=145.79.227.28:4731 , piece_id=EPW3LUA3EWRCURHZ66NCRCA36CNIW4GYQMFNKPZZTRCB3YBGTKJA, seg_pos={0 0}, upload=473490]: Long tail cancellation due to optimal threshold
Dec 28 03:23:38.816128436 (539h54m38.687211481s) [seg_pos={0 0}, upload=473490]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 28 03:23:38.816257582 (539h54m38.687340627s) [upload=473490]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 03:23:38.816265778 (539h54m38.687348813s) [upload=473490]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 03:23:39.111115407 (539h54m38.982198442s) [upload=473490]: Done uploading segments...
Dec 28 03:23:39.111138972 (539h54m38.982222008s) [upload=473490]: Upload finished. err: <nil>
Duration: 56.72426146s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
19 out of 60 is failed (threshold=6) [.....X.....XX.....X..X.X....X.XXXX....X.X.X...X..X.X.X..X...] duration:
3.589110651s trace:
https://jaeger.dev.storj.io/trace/15610cbc45f91fb8
FAIL (2025-12-28 03:50:52 UTC)
orbiter warning: context deadline exceeded
Duration: 3.589139276s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
50 out of 60 is failed (threshold=3) [X.X..XX..XXXXXX.XXXXXXXXXXXXXXXXX.XXXXX.XXXXXXXXXXXXXXXXX..X] duration:
30.002768432s trace:
https://jaeger.dev.storj.io/trace/35a7d908c6f66f7a
FAIL (2025-12-28 03:50:48 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: LV4JGPOX4IOTTZNFLUQ4FDWNOUG56X33Z5V6PJ3SCXTCX3AL4NPA) context deadline exceeded; piecestore: (Node ID: 1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, Piece ID: E2XQVMKZGBPCF42QHN7ODPTIIAC3FZLVGXO4FHVROC27XRZHHTMQ) context deadline exceeded
======LOGS======
50:18.920265041 (540h21m18.791348086s) [upload=473885]: Starting upload
Dec 28 03:50:18.920345895 (540h21m18.79142893s) [upload=473885]: Done starting upload
Dec 28 03:50:18.920635958 (540h21m18.791718993s) [upload=473885]: Uploading segments...
Dec 28 03:50:18.920688207 (540h21m18.791771242s) [upload=473885]: Got next segment. Inline: false
Dec 28 03:50:18.920720509 (540h21m18.791803544s) [upload=473885]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 28 03:50:18.920726289 (540h21m18.791809324s) [upload=473885]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 28 03:50:19.170637694 (540h21m19.041720749s) [seg_pos={0 0}, upload=473885]: Begin upload segment...
Dec 28 03:50:19.170709540 (540h21m19.041792596s) [seg_pos={0 0}, upload=473885]: Done begin upload segment.
Dec 28 03:50:19.170776238 (540h21m19.041859273s) [noise=true , address=145.79.227.5:4221 , piece_id=E4XN5VOBLELFQ2LT7DRW3JUJXNCG27GK4TLQGVVLZA4P6UGAGG4A, seg_pos={0 0}, upload=473885]: Uploading piece...
Dec 28 03:50:19.170794352 (540h21m19.041877397s) [noise=true , address=45.91.116.6:4975 , piece_id=LV4JGPOX4IOTTZNFLUQ4FDWNOUG56X33Z5V6PJ3SCXTCX3AL4NPA, seg_pos={0 0}, upload=473885]: Uploading piece...
Dec 28 03:50:19.171485037 (540h21m19.042568082s) [noise=true , address=145.79.227.47:4267 , piece_id=5O3DM2OE6A4YBI52G3WTKFK2M3IKLD2IZ7RBTSL4RLHEDNH2D4DA, seg_pos={0 0}, upload=473885]: Uploading piece...
Dec 28 03:50:19.171757366 (540h21m19.042840401s) [noise=true , address=145.79.227.46:4603 , piece_id=CMJU54TNROYFPKLYZWV4QYGC36NSSX6FSLNRK3RIKU2MAEKK6G2Q, seg_pos={0 0}, upload=473885]: Uploading piece...
Dec 28 03:50:19.172068658 (540h21m19.043151693s) [noise=true , address=77.90.31.18:4808 , piece_id=DJJBKEDNAAB5MY4ZOGYJNOEABQU32VICWSHPZYRSOPXZB55ILEXQ, seg_pos={0 0}, upload=473885]: Uploading piece...
Dec 28 03:50:19.172138301 (540h21m19.043221346s) [noise=true , address=145.79.227.33:4871 , piece_id=BQBANGJUXEWLSU4642A5X7PSLW4SFITHUZQADN3ZX5M24P6S4XDQ, seg_pos={0 0}, upload=473885]: Uploading piece...
Dec 28 03:50:19.172730378 (540h21m19.043813434s) [noise=true , address=145.79.227.48:4302 , piece_id=C6USMXK2GE6ACHL6AKJM7HTNXNV63SRUUQPFR6GRI3HFWGTH7RTQ, seg_pos={0 0}, upload=473885]: Uploading piece...
Dec 28 03:50:19.172912906 (540h21m19.043995951s) [noise=true , address=145.79.227.15:4874 , piece_id=6Q3PJGJLGF47MANF2RVDDLPIKLP256TR7OUCS6GVEAD2ODAMEFPA, seg_pos={0 0}, upload=473885]: Uploading piece...
Dec 28 03:50:19.173246742 (540h21m19.044329777s) [noise=true , address=45.91.116.4:4937 , piece_id=UKLI4CBDBMJKVJLPWE3SS3ICJJ23CDG44MTQHFTDREQ2BWG5I5PQ, seg_pos={0 0}, upload=473885]: Uploading piece...
Dec 28 03:50:19.173531093 (540h21m19.044614138s) [noise=true , address=145.79.227.14:4839 , piece_id=43AEPWOYYDH32GPB2EMV4YDXKKSRNJBL3VM6YLDCMQ4HBVO77KKA, seg_pos={0 0}, upload=473885]: Uploading piece...
Dec 28 03:50:19.173888974 (540h21m19.044972019s) [noise=true , address=45.91.116.8:4047 , piece_id=GLVOIFVR5LC4GK6DDF4HIJ2MHEMF26D7BUBCADYPZNT65HVBOVKQ, seg_pos={0 0}, upload=473885]: Uploading piece...
Dec 28 03:50:19.174244121 (540h21m19.045327186s) [noise=true , address=145.79.227.43:4157 , piece_id=G53XMP265NN25DTO3Y3VXMO7ZYL7H62GOZM3SHC7GM6RE445STKA, seg_pos={0 0}, upload=473885]: Uploading piece...
Dec 28 03:50:19.174493516 (540h21m19.045576651s) [noise=true , address=77.90.31.16:4736 , piece_id=E2XQVMKZGBPCF42QHN7ODPTIIAC3FZLVGXO4FHVROC27XRZHHTMQ, seg_pos={0 0}, upload=473885]: Uploading piece...
Dec 28 03:50:19.175125379 (540h21m19.046208484s) [noise=true , address=145.79.227.51:4411 , piece_id=SW4DTGBCOHQGKNUN5IV6Q6ZYVPSD2GZP3DF7EHCYNI6M4X5Y4OQQ, seg_pos={0 0}, upload=473885]: Uploading piece...
Dec 28 03:50:19.175137762 (540h21m19.046220807s) [noise=true , address=145.79.227.11:4730 , piece_id=FLIHULO2KI34JBAMXPJSRFD2N6TEQVLM3QVFEI6HWX4TOLVR5JWA, seg_pos={0 0}, upload=473885]: Uploading piece...
Dec 28 03:50:19.175730241 (540h21m19.046813276s) [noise=true , address=45.91.116.9:4261 , piece_id=4VVTLV7J3ONKGFJCXTLX5DEI2CVA2OEMGMGX2K4KDFXS56LZFBAQ, seg_pos={0 0}, upload=473885]: Uploading piece...
Dec 28 03:50:20.803781039 (540h21m20.674864084s) [upload=473885]: Next returned nil segment
Dec 28 03:50:20.803798082 (540h21m20.674881117s) [upload=473885]: Waiting for error group managing segments...
Dec 28 03:50:21.192492021 (540h21m21.063575066s) [noise=true , address=145.79.227.15:4874 , piece_id=6Q3PJGJLGF47MANF2RVDDLPIKLP256TR7OUCS6GVEAD2ODAMEFPA, seg_pos={0 0}, upload=473885]: Done uploading piece. err: <nil>
Dec 28 03:50:21.195832782 (540h21m21.066915818s) [noise=true , address=145.79.227.33:4871 , piece_id=BQBANGJUXEWLSU4642A5X7PSLW4SFITHUZQADN3ZX5M24P6S4XDQ, seg_pos={0 0}, upload=473885]: Done uploading piece. err: <nil>
Dec 28 03:50:21.201248234 (540h21m21.072331279s) [noise=true , address=145.79.227.11:4730 , piece_id=FLIHULO2KI34JBAMXPJSRFD2N6TEQVLM3QVFEI6HWX4TOLVR5JWA, seg_pos={0 0}, upload=473885]: Done uploading piece. err: <nil>
Dec 28 03:50:21.211557636 (540h21m21.082640681s) [noise=true , address=145.79.227.5:4221 , piece_id=E4XN5VOBLELFQ2LT7DRW3JUJXNCG27GK4TLQGVVLZA4P6UGAGG4A, seg_pos={0 0}, upload=473885]: Done uploading piece. err: <nil>
Dec 28 03:50:21.214773539 (540h21m21.085856584s) [noise=true , address=145.79.227.14:4839 , piece_id=43AEPWOYYDH32GPB2EMV4YDXKKSRNJBL3VM6YLDCMQ4HBVO77KKA, seg_pos={0 0}, upload=473885]: Done uploading piece. err: <nil>
Dec 28 03:50:21.215389592 (540h21m21.086472627s) [noise=true , address=145.79.227.46:4603 , piece_id=CMJU54TNROYFPKLYZWV4QYGC36NSSX6FSLNRK3RIKU2MAEKK6G2Q, seg_pos={0 0}, upload=473885]: Done uploading piece. err: <nil>
Dec 28 03:50:21.236537411 (540h21m21.107620456s) [noise=true , address=145.79.227.47:4267 , piece_id=5O3DM2OE6A4YBI52G3WTKFK2M3IKLD2IZ7RBTSL4RLHEDNH2D4DA, seg_pos={0 0}, upload=473885]: Done uploading piece. err: <nil>
Dec 28 03:50:21.460545010 (540h21m21.331628045s) [noise=true , address=77.90.31.16:4736 , piece_id=E2XQVMKZGBPCF42QHN7ODPTIIAC3FZLVGXO4FHVROC27XRZHHTMQ, seg_pos={0 0}, upload=473885]: Done uploading piece. err: <nil>
Dec 28 03:50:21.523577044 (540h21m21.394660089s) [noise=true , address=45.91.116.4:4937 , piece_id=UKLI4CBDBMJKVJLPWE3SS3ICJJ23CDG44MTQHFTDREQ2BWG5I5PQ, seg_pos={0 0}, upload=473885]: Done uploading piece. err: <nil>
Dec 28 03:50:21.567604159 (540h21m21.438687194s) [noise=true , address=145.79.227.51:4411 , piece_id=SW4DTGBCOHQGKNUN5IV6Q6ZYVPSD2GZP3DF7EHCYNI6M4X5Y4OQQ, seg_pos={0 0}, upload=473885]: Done uploading piece. err: <nil>
Dec 28 03:50:21.569622612 (540h21m21.440705647s) [noise=true , address=45.91.116.6:4975 , piece_id=LV4JGPOX4IOTTZNFLUQ4FDWNOUG56X33Z5V6PJ3SCXTCX3AL4NPA, seg_pos={0 0}, upload=473885]: Done uploading piece. err: <nil>
Dec 28 03:50:21.803745953 (540h21m21.674828998s) [noise=true , address=45.91.116.9:4261 , piece_id=4VVTLV7J3ONKGFJCXTLX5DEI2CVA2OEMGMGX2K4KDFXS56LZFBAQ, seg_pos={0 0}, upload=473885]: Done uploading piece. err: <nil>
Dec 28 03:50:21.861746777 (540h21m21.732829812s) [noise=true , address=77.90.31.18:4808 , piece_id=DJJBKEDNAAB5MY4ZOGYJNOEABQU32VICWSHPZYRSOPXZB55ILEXQ, seg_pos={0 0}, upload=473885]: Done uploading piece. err: <nil>
Dec 28 03:50:21.861768308 (540h21m21.732851343s) [seg_pos={0 0}, upload=473885]: Segment reached optimal threshold of 13 pieces.
Dec 28 03:50:21.861955715 (540h21m21.73303875s) [noise=true , address=145.79.227.48:4302 , piece_id=C6USMXK2GE6ACHL6AKJM7HTNXNV63SRUUQPFR6GRI3HFWGTH7RTQ, seg_pos={0 0}, upload=473885]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tGpuYdPLiy4ncDjxrZbQfp7Bq79NVBHDmr4vjaL4qMpudrvgR): protocol: write tcp 188.166.225.170:60302->145.79.227.48:4302: use of closed network connection; write tcp 188.166.225.170:60302->145.79.227.48:4302: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60302->145.79.227.48:4302: use of closed network connection
Dec 28 03:50:21.861961486 (540h21m21.733044521s) [noise=true , address=145.79.227.48:4302 , piece_id=C6USMXK2GE6ACHL6AKJM7HTNXNV63SRUUQPFR6GRI3HFWGTH7RTQ, seg_pos={0 0}, upload=473885]: Long tail cancellation due to optimal threshold
Dec 28 03:50:21.862228004 (540h21m21.733311049s) [noise=true , address=145.79.227.43:4157 , piece_id=G53XMP265NN25DTO3Y3VXMO7ZYL7H62GOZM3SHC7GM6RE445STKA, seg_pos={0 0}, upload=473885]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZrzADk2mKG4R9tjxUKjQeJvb4WmfjLksGFyvstHRfYbBC7XNz): protocol: expected piece hash; context canceled; EOF
Dec 28 03:50:21.862238203 (540h21m21.733321238s) [noise=true , address=145.79.227.43:4157 , piece_id=G53XMP265NN25DTO3Y3VXMO7ZYL7H62GOZM3SHC7GM6RE445STKA, seg_pos={0 0}, upload=473885]: Long tail cancellation due to optimal threshold
Dec 28 03:50:21.862402275 (540h21m21.733485311s) [noise=true , address=45.91.116.8:4047 , piece_id=GLVOIFVR5LC4GK6DDF4HIJ2MHEMF26D7BUBCADYPZNT65HVBOVKQ, seg_pos={0 0}, upload=473885]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF
Dec 28 03:50:21.862411884 (540h21m21.733494929s) [noise=true , address=45.91.116.8:4047 , piece_id=GLVOIFVR5LC4GK6DDF4HIJ2MHEMF26D7BUBCADYPZNT65HVBOVKQ, seg_pos={0 0}, upload=473885]: Long tail cancellation due to optimal threshold
Dec 28 03:50:21.862430829 (540h21m21.733513864s) [seg_pos={0 0}, upload=473885]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 28 03:50:21.862514379 (540h21m21.733597414s) [upload=473885]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 03:50:21.862521402 (540h21m21.733604447s) [upload=473885]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 03:50:22.152221390 (540h21m22.023304435s) [upload=473885]: Done uploading segments...
Dec 28 03:50:22.152245215 (540h21m22.02332825s) [upload=473885]: Upload finished. err: <nil>
Duration: 30.002792899s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
19 out of 60 is failed (threshold=6) [......X.....XX.....X..X.X....X.XXXX....X.X.X...X..X.X.X..X..] duration:
3.496236093s trace:
https://jaeger.dev.storj.io/trace/15610cbc45f91fb8
FAIL (2025-12-28 03:50:36 UTC)
orbiter warning: context deadline exceeded
Duration: 3.49626629s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..............................................X.............] duration:
12.354391258s trace:
https://jaeger.dev.storj.io/trace/3b8130f32e60318d
FAIL (2025-12-28 03:50:25 UTC)
orbiter warning: context deadline exceeded
Duration: 12.354425663s
edge
gateway
us-select-1
|