Orbiter: EU/US/AP/EDGE | Code | Config | Playbook | Escalation Procedure
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..X.........................................................] duration:
3.213702746s trace:
https://jaeger.dev.storj.io/trace/2fe350ab9c48d459
FAIL (2026-04-19 14:53:41 UTC)
orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 23)
Duration: 3.213793731s
core
uplink
ap1
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...X........................................................] duration:
3.117410559s trace:
https://jaeger.dev.storj.io/trace/2fe350ab9c48d459
FAIL (2026-04-19 14:53:25 UTC)
orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 23)
Duration: 3.117435799s
core
uplink
ap1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [.....X......................................................] duration:
3.49017916s trace:
https://jaeger.dev.storj.io/trace/3c23449320be8b26
FAIL (2026-04-19 14:53:02 UTC)
orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded
Duration: 3.490204769s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.X..........................................................] duration:
5.407023408s trace:
https://jaeger.dev.storj.io/trace/59b3e23b0ad9afe7
FAIL (2026-04-19 14:52:49 UTC)
orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 39)
Duration: 5.407043217s
core
uplink
us1
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.remote)antiblip:
1 out of 60 is failed (threshold=3) [....X.......................................................] duration:
3.552222479s trace:
https://jaeger.dev.storj.io/trace/2fe350ab9c48d459
FAIL (2026-04-19 14:52:44 UTC)
orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 23)
Duration: 3.552249382s
core
uplink
ap1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..X.........................................................] duration:
5.514830991s trace:
https://jaeger.dev.storj.io/trace/59b3e23b0ad9afe7
FAIL (2026-04-19 14:52:26 UTC)
orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 39)
Duration: 5.514860709s
core
uplink
us1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [......X.....................................................] duration:
3.529827651s trace:
https://jaeger.dev.storj.io/trace/3c23449320be8b26
FAIL (2026-04-19 14:52:26 UTC)
orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded
Duration: 3.529895392s
core
uplink
us-select-1
us1-placement-20
|
checkers.HTTP{"GET" "https://www.apple.com/robots.txt"} failure (antiblip window 3/60): (http.apple_robots_txt)antiblip:
1 out of 60 is failed (threshold=3) [X...........................................................] duration:
31.508308ms remote:
"tcp:104.119.104.223:443" target:
https://www.apple.com/robots.txt trace:
https://jaeger.dev.storj.io/trace/505bb4ef80a6b4b1
FAIL (2026-04-19 14:52:04 UTC)
orbiter warning: Get "https://www.apple.com/robots.txt": EOF
Duration: 31.546121ms
core
website
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [X...........................................................] duration:
5.732579866s trace:
https://jaeger.dev.storj.io/trace/2e2125521d7abc3e
FAIL (2026-04-19 14:51:56 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 5)
======LOGS======
e...
Apr 19 14:08:55.493690567 (286h22m42.312926591s) [noise=true , address=77.90.31.16:4738 , piece_id=NOF6IEE7PK5DHZ7FMWTRGL6WJ5I7D5XK7HEZKEUB7SCV6YGW3JQQ, seg_pos={0 0}, upload=274918]: Uploading piece...
Apr 19 14:08:55.494428361 (286h22m42.313664385s) [noise=true , address=145.79.227.13:4805 , piece_id=GGNMTIQPEMO7RN6KJFGCOUU6XX65CKHFJIZHFC2JVYR2JNJK3Y6Q, seg_pos={0 0}, upload=274918]: Uploading piece...
Apr 19 14:08:55.494992880 (286h22m42.314228904s) [noise=true , address=45.91.116.12:4227 , piece_id=LODLVJMI2EAMJ3EAAOI4Z4RWMOAQHFZL45MSWKVS5MB7FCNJ57AA, seg_pos={0 0}, upload=274918]: Uploading piece...
Apr 19 14:08:55.495362383 (286h22m42.314598407s) [noise=true , address=145.79.227.51:4415 , piece_id=RDAXQUREP4SXGB4TCQJUXD6YPRXXBFJIRFEPHUOA7ZYX6A5QOM2A, seg_pos={0 0}, upload=274918]: Uploading piece...
Apr 19 14:08:55.495930219 (286h22m42.315166243s) [noise=true , address=45.91.116.11:4152 , piece_id=O46QV2WY5ZRZSU3BJK32J6B2QAYIQH6VPSDB5LVZEHIL6FITZQQQ, seg_pos={0 0}, upload=274918]: Uploading piece...
Apr 19 14:08:55.496278582 (286h22m42.315514606s) [noise=true , address=145.79.227.15:4879 , piece_id=MWHLQQFFDK2FDKJN7J4MY7FY5O3MB6WMQFXBOLTXMRPRCTSVMQRQ, seg_pos={0 0}, upload=274918]: Uploading piece...
Apr 19 14:08:55.497181003 (286h22m42.316417047s) [noise=true , address=145.79.227.9:4366 , piece_id=D2KZ4TICJOQGO4FN5ITDMYVH6NH7FAFDKVNDYZTFSGDNGO3AT4FQ, seg_pos={0 0}, upload=274918]: Uploading piece...
Apr 19 14:08:55.497590223 (286h22m42.316826258s) [noise=true , address=145.79.227.34:4907 , piece_id=OVWJEETOWMYMW4OBXTEM2IR44YU2USTJHSKVONV4FMV2LOTFXM5Q, seg_pos={0 0}, upload=274918]: Uploading piece...
Apr 19 14:08:55.498297648 (286h22m42.317533672s) [noise=true , address=145.79.227.36:4518 , piece_id=JC5WKYW543ZUQOAJG33PLM4A6A7WYGWBC6QLYCUKEGSSHYX6JDBA, seg_pos={0 0}, upload=274918]: Uploading piece...
Apr 19 14:08:55.498945970 (286h22m42.318182014s) [noise=true , address=145.79.227.38:4982 , piece_id=ERRKLW4B4WI7YWVD2JJPEDHSU4IEJJM5UHFHTJ3CELIWXNZTJ34Q, seg_pos={0 0}, upload=274918]: Uploading piece...
Apr 19 14:08:55.499381660 (286h22m42.318617684s) [noise=true , address=145.79.227.2:4119 , piece_id=ME7PKNTEYNW36GBEP2LNSMUZMTS7JR67BEUHWEDYTWKB6ANYYHKQ, seg_pos={0 0}, upload=274918]: Uploading piece...
Apr 19 14:08:55.500079246 (286h22m42.31931527s) [noise=true , address=145.79.227.37:4568 , piece_id=XT3EGREWAC2Q6RZXKJAM6ORKDYUSNRIKT5RJOMGEJ4B2N2PEC46A, seg_pos={0 0}, upload=274918]: Uploading piece...
Apr 19 14:08:55.500330311 (286h22m42.319566335s) [noise=true , address=77.90.31.15:4700 , piece_id=CGWM3PZIGR2LX63KTTNJALEFI4NY2YVLBAZAXKGK2BE3V37D6GMA, seg_pos={0 0}, upload=274918]: Uploading piece...
Apr 19 14:08:55.501048167 (286h22m42.320284191s) [noise=true , address=145.79.227.3:4154 , piece_id=GCLCPG2QCIT4AMKHMAU4QZHPAONPDNSJAGLLVN4TO2VBZZGJOGIA, seg_pos={0 0}, upload=274918]: Uploading piece...
Apr 19 14:08:55.501263162 (286h22m42.320499186s) [noise=true , address=145.79.227.17:4972 , piece_id=57NAU5R6YCE4AA3PLAKII2U4K5KJCYDS3NH5PZAOUL36W7MSLONQ, seg_pos={0 0}, upload=274918]: Uploading piece...
Apr 19 14:08:57.949098467 (286h22m44.768334501s) [upload=274918]: Next returned nil segment
Apr 19 14:08:57.949115480 (286h22m44.768351554s) [upload=274918]: Waiting for error group managing segments...
Apr 19 14:08:58.345206270 (286h22m45.164442294s) [noise=true , address=145.79.227.3:4154 , piece_id=GCLCPG2QCIT4AMKHMAU4QZHPAONPDNSJAGLLVN4TO2VBZZGJOGIA, seg_pos={0 0}, upload=274918]: Done uploading piece. err: <nil>
Apr 19 14:09:01.082767579 (286h22m47.902003603s) [noise=true , address=45.91.116.11:4152 , piece_id=O46QV2WY5ZRZSU3BJK32J6B2QAYIQH6VPSDB5LVZEHIL6FITZQQQ, seg_pos={0 0}, upload=274918]: Done uploading piece. err: <nil>
Apr 19 14:09:05.145686115 (286h22m51.964922139s) [noise=true , address=77.90.31.16:4738 , piece_id=NOF6IEE7PK5DHZ7FMWTRGL6WJ5I7D5XK7HEZKEUB7SCV6YGW3JQQ, seg_pos={0 0}, upload=274918]: Done uploading piece. err: <nil>
Apr 19 14:09:10.080594511 (286h22m56.899830535s) [noise=true , address=145.79.227.38:4982 , piece_id=ERRKLW4B4WI7YWVD2JJPEDHSU4IEJJM5UHFHTJ3CELIWXNZTJ34Q, seg_pos={0 0}, upload=274918]: Done uploading piece. err: <nil>
Apr 19 14:09:11.441874037 (286h22m58.261110061s) [noise=true , address=145.79.227.15:4879 , piece_id=MWHLQQFFDK2FDKJN7J4MY7FY5O3MB6WMQFXBOLTXMRPRCTSVMQRQ, seg_pos={0 0}, upload=274918]: Done uploading piece. err: <nil>
Apr 19 14:09:24.600269009 (286h23m11.419505053s) [noise=true , address=145.79.227.2:4119 , piece_id=ME7PKNTEYNW36GBEP2LNSMUZMTS7JR67BEUHWEDYTWKB6ANYYHKQ, seg_pos={0 0}, upload=274918]: Done uploading piece. err: ecclient: upload failed (node:121xNAnkm9HZXUszp26UmWTsuhp3igyeDVea6gSCdtrQUQDVHpW, address:145.79.227.2:4119): protocol: expected piece hash; context deadline exceeded; EOF
Apr 19 14:09:24.600415442 (286h23m11.419651496s) [noise=true , address=145.79.227.17:4972 , piece_id=57NAU5R6YCE4AA3PLAKII2U4K5KJCYDS3NH5PZAOUL36W7MSLONQ, seg_pos={0 0}, upload=274918]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: write tcp 188.166.225.170:55998->145.79.227.17:4972: use of closed network connection; write tcp 188.166.225.170:55998->145.79.227.17:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55998->145.79.227.17:4972: use of closed network connection
Apr 19 14:09:24.600668150 (286h23m11.419904204s) [noise=true , address=145.79.227.9:4366 , piece_id=D2KZ4TICJOQGO4FN5ITDMYVH6NH7FAFDKVNDYZTFSGDNGO3AT4FQ, seg_pos={0 0}, upload=274918]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: rpc: dial tcp 145.79.227.9:4366: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 19 14:09:24.600698107 (286h23m11.419934181s) [noise=true , address=145.79.227.36:4518 , piece_id=JC5WKYW543ZUQOAJG33PLM4A6A7WYGWBC6QLYCUKEGSSHYX6JDBA, seg_pos={0 0}, upload=274918]: Done uploading piece. err: ecclient: upload failed (node:17TZ9JLdCivFF46guhSTW6hcoUmTz5uo5wvLkDqC4BGmpm9eLb, address:145.79.227.36:4518): protocol: expected piece hash; context deadline exceeded; EOF
Apr 19 14:09:24.600963510 (286h23m11.420199584s) [noise=true , address=77.90.31.15:4700 , piece_id=CGWM3PZIGR2LX63KTTNJALEFI4NY2YVLBAZAXKGK2BE3V37D6GMA, seg_pos={0 0}, upload=274918]: Done uploading piece. err: ecclient: upload failed (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, address:77.90.31.15:4700): protocol: read tcp 188.166.225.170:48972->77.90.31.15:4700: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 19 14:09:24.601113419 (286h23m11.420349453s) [noise=true , address=145.79.227.34:4907 , piece_id=OVWJEETOWMYMW4OBXTEM2IR44YU2USTJHSKVONV4FMV2LOTFXM5Q, seg_pos={0 0}, upload=274918]: Done uploading piece. err: ecclient: upload failed (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS, address:145.79.227.34:4907): protocol: rpc: dial tcp 145.79.227.34:4907: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 19 14:09:24.601152354 (286h23m11.420388378s) [noise=true , address=45.91.116.7:4012 , piece_id=SDT4HR32F6WKFNFJXHMU4KMUGXUEJ32OYUPOC6SJBTATEXUL5RNQ, seg_pos={0 0}, upload=274918]: Done uploading piece. err: ecclient: upload failed (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, address:45.91.116.7:4012): protocol: read tcp 188.166.225.170:41998->45.91.116.7:4012: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 19 14:09:24.601398209 (286h23m11.420634233s) [noise=true , address=145.79.227.51:4415 , piece_id=RDAXQUREP4SXGB4TCQJUXD6YPRXXBFJIRFEPHUOA7ZYX6A5QOM2A, seg_pos={0 0}, upload=274918]: Done uploading piece. err: ecclient: upload failed (node:129sUhRR5RYtmrrfcEq2KgikZqZqQMRbwgK8NAKLFTTZgXDHW1v, address:145.79.227.51:4415): protocol: read tcp 188.166.225.170:59672->145.79.227.51:4415: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 19 14:09:24.601602834 (286h23m11.420838888s) [noise=true , address=145.79.227.37:4568 , piece_id=XT3EGREWAC2Q6RZXKJAM6ORKDYUSNRIKT5RJOMGEJ4B2N2PEC46A, seg_pos={0 0}, upload=274918]: Done uploading piece. err: ecclient: upload failed (node:1zJigHghNntzHNWim5RnWcMPfeUAbqsrvrfWretYcyEkXbSYzJ, address:145.79.227.37:4568): protocol: read tcp 188.166.225.170:49018->145.79.227.37:4568: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 19 14:09:24.601856011 (286h23m11.421092045s) [noise=true , address=145.79.227.13:4805 , piece_id=GGNMTIQPEMO7RN6KJFGCOUU6XX65CKHFJIZHFC2JVYR2JNJK3Y6Q, seg_pos={0 0}, upload=274918]: Done uploading piece. err: ecclient: upload failed (node:12YEVsEwZwfWsnZFyaSkd9Dn53Tb9EFUyWEVDwTQeMJkcjwrj7x, address:145.79.227.13:4805): protocol: read tcp 188.166.225.170:46898->145.79.227.13:4805: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 19 14:09:24.601963369 (286h23m11.421199393s) [noise=true , address=45.91.116.12:4227 , piece_id=LODLVJMI2EAMJ3EAAOI4Z4RWMOAQHFZL45MSWKVS5MB7FCNJ57AA, seg_pos={0 0}, upload=274918]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: read tcp 188.166.225.170:44102->45.91.116.12:4227: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 19 14:09:24.602008025 (286h23m11.42124405s) [seg_pos={0 0}, upload=274918]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil>
Apr 19 14:09:24.851854994 (286h23m11.671091018s) [upload=274918]: Done uploading segments...
Apr 19 14:09:24.851892576 (286h23m11.6711286s) [upload=274918]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5)
Duration: 5.732612469s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.....X......................................................] duration:
3.008395058s trace:
https://jaeger.dev.storj.io/trace/2fe350ab9c48d459
FAIL (2026-04-19 14:51:41 UTC)
orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 23)
Duration: 3.008427359s
core
uplink
ap1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...X........................................................] duration:
5.527231014s trace:
https://jaeger.dev.storj.io/trace/59b3e23b0ad9afe7
FAIL (2026-04-19 14:51:36 UTC)
orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 39)
Duration: 5.527263476s
core
uplink
us1
|
checkers.HTTP{"GET" "https://www.apple.com/robots.txt"} failure (antiblip window 3/60): (http.apple_robots_txt)antiblip:
1 out of 60 is failed (threshold=3) [.X..........................................................] duration:
16.201689ms remote:
"tcp:104.119.104.223:443" target:
https://www.apple.com/robots.txt trace:
https://jaeger.dev.storj.io/trace/505bb4ef80a6b4b1
FAIL (2026-04-19 14:51:32 UTC)
orbiter warning: Get "https://www.apple.com/robots.txt": EOF
Duration: 16.222118ms
core
website
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [.......X....................................................] duration:
3.490708081s trace:
https://jaeger.dev.storj.io/trace/3c23449320be8b26
FAIL (2026-04-19 14:51:22 UTC)
orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded
Duration: 3.490732908s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.X..........................................................] duration:
5.762995173s trace:
https://jaeger.dev.storj.io/trace/2e2125521d7abc3e
FAIL (2026-04-19 14:51:16 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 5)
======LOGS======
e...
Apr 19 14:08:55.493690567 (286h22m42.312926591s) [noise=true , address=77.90.31.16:4738 , piece_id=NOF6IEE7PK5DHZ7FMWTRGL6WJ5I7D5XK7HEZKEUB7SCV6YGW3JQQ, seg_pos={0 0}, upload=274918]: Uploading piece...
Apr 19 14:08:55.494428361 (286h22m42.313664385s) [noise=true , address=145.79.227.13:4805 , piece_id=GGNMTIQPEMO7RN6KJFGCOUU6XX65CKHFJIZHFC2JVYR2JNJK3Y6Q, seg_pos={0 0}, upload=274918]: Uploading piece...
Apr 19 14:08:55.494992880 (286h22m42.314228904s) [noise=true , address=45.91.116.12:4227 , piece_id=LODLVJMI2EAMJ3EAAOI4Z4RWMOAQHFZL45MSWKVS5MB7FCNJ57AA, seg_pos={0 0}, upload=274918]: Uploading piece...
Apr 19 14:08:55.495362383 (286h22m42.314598407s) [noise=true , address=145.79.227.51:4415 , piece_id=RDAXQUREP4SXGB4TCQJUXD6YPRXXBFJIRFEPHUOA7ZYX6A5QOM2A, seg_pos={0 0}, upload=274918]: Uploading piece...
Apr 19 14:08:55.495930219 (286h22m42.315166243s) [noise=true , address=45.91.116.11:4152 , piece_id=O46QV2WY5ZRZSU3BJK32J6B2QAYIQH6VPSDB5LVZEHIL6FITZQQQ, seg_pos={0 0}, upload=274918]: Uploading piece...
Apr 19 14:08:55.496278582 (286h22m42.315514606s) [noise=true , address=145.79.227.15:4879 , piece_id=MWHLQQFFDK2FDKJN7J4MY7FY5O3MB6WMQFXBOLTXMRPRCTSVMQRQ, seg_pos={0 0}, upload=274918]: Uploading piece...
Apr 19 14:08:55.497181003 (286h22m42.316417047s) [noise=true , address=145.79.227.9:4366 , piece_id=D2KZ4TICJOQGO4FN5ITDMYVH6NH7FAFDKVNDYZTFSGDNGO3AT4FQ, seg_pos={0 0}, upload=274918]: Uploading piece...
Apr 19 14:08:55.497590223 (286h22m42.316826258s) [noise=true , address=145.79.227.34:4907 , piece_id=OVWJEETOWMYMW4OBXTEM2IR44YU2USTJHSKVONV4FMV2LOTFXM5Q, seg_pos={0 0}, upload=274918]: Uploading piece...
Apr 19 14:08:55.498297648 (286h22m42.317533672s) [noise=true , address=145.79.227.36:4518 , piece_id=JC5WKYW543ZUQOAJG33PLM4A6A7WYGWBC6QLYCUKEGSSHYX6JDBA, seg_pos={0 0}, upload=274918]: Uploading piece...
Apr 19 14:08:55.498945970 (286h22m42.318182014s) [noise=true , address=145.79.227.38:4982 , piece_id=ERRKLW4B4WI7YWVD2JJPEDHSU4IEJJM5UHFHTJ3CELIWXNZTJ34Q, seg_pos={0 0}, upload=274918]: Uploading piece...
Apr 19 14:08:55.499381660 (286h22m42.318617684s) [noise=true , address=145.79.227.2:4119 , piece_id=ME7PKNTEYNW36GBEP2LNSMUZMTS7JR67BEUHWEDYTWKB6ANYYHKQ, seg_pos={0 0}, upload=274918]: Uploading piece...
Apr 19 14:08:55.500079246 (286h22m42.31931527s) [noise=true , address=145.79.227.37:4568 , piece_id=XT3EGREWAC2Q6RZXKJAM6ORKDYUSNRIKT5RJOMGEJ4B2N2PEC46A, seg_pos={0 0}, upload=274918]: Uploading piece...
Apr 19 14:08:55.500330311 (286h22m42.319566335s) [noise=true , address=77.90.31.15:4700 , piece_id=CGWM3PZIGR2LX63KTTNJALEFI4NY2YVLBAZAXKGK2BE3V37D6GMA, seg_pos={0 0}, upload=274918]: Uploading piece...
Apr 19 14:08:55.501048167 (286h22m42.320284191s) [noise=true , address=145.79.227.3:4154 , piece_id=GCLCPG2QCIT4AMKHMAU4QZHPAONPDNSJAGLLVN4TO2VBZZGJOGIA, seg_pos={0 0}, upload=274918]: Uploading piece...
Apr 19 14:08:55.501263162 (286h22m42.320499186s) [noise=true , address=145.79.227.17:4972 , piece_id=57NAU5R6YCE4AA3PLAKII2U4K5KJCYDS3NH5PZAOUL36W7MSLONQ, seg_pos={0 0}, upload=274918]: Uploading piece...
Apr 19 14:08:57.949098467 (286h22m44.768334501s) [upload=274918]: Next returned nil segment
Apr 19 14:08:57.949115480 (286h22m44.768351554s) [upload=274918]: Waiting for error group managing segments...
Apr 19 14:08:58.345206270 (286h22m45.164442294s) [noise=true , address=145.79.227.3:4154 , piece_id=GCLCPG2QCIT4AMKHMAU4QZHPAONPDNSJAGLLVN4TO2VBZZGJOGIA, seg_pos={0 0}, upload=274918]: Done uploading piece. err: <nil>
Apr 19 14:09:01.082767579 (286h22m47.902003603s) [noise=true , address=45.91.116.11:4152 , piece_id=O46QV2WY5ZRZSU3BJK32J6B2QAYIQH6VPSDB5LVZEHIL6FITZQQQ, seg_pos={0 0}, upload=274918]: Done uploading piece. err: <nil>
Apr 19 14:09:05.145686115 (286h22m51.964922139s) [noise=true , address=77.90.31.16:4738 , piece_id=NOF6IEE7PK5DHZ7FMWTRGL6WJ5I7D5XK7HEZKEUB7SCV6YGW3JQQ, seg_pos={0 0}, upload=274918]: Done uploading piece. err: <nil>
Apr 19 14:09:10.080594511 (286h22m56.899830535s) [noise=true , address=145.79.227.38:4982 , piece_id=ERRKLW4B4WI7YWVD2JJPEDHSU4IEJJM5UHFHTJ3CELIWXNZTJ34Q, seg_pos={0 0}, upload=274918]: Done uploading piece. err: <nil>
Apr 19 14:09:11.441874037 (286h22m58.261110061s) [noise=true , address=145.79.227.15:4879 , piece_id=MWHLQQFFDK2FDKJN7J4MY7FY5O3MB6WMQFXBOLTXMRPRCTSVMQRQ, seg_pos={0 0}, upload=274918]: Done uploading piece. err: <nil>
Apr 19 14:09:24.600269009 (286h23m11.419505053s) [noise=true , address=145.79.227.2:4119 , piece_id=ME7PKNTEYNW36GBEP2LNSMUZMTS7JR67BEUHWEDYTWKB6ANYYHKQ, seg_pos={0 0}, upload=274918]: Done uploading piece. err: ecclient: upload failed (node:121xNAnkm9HZXUszp26UmWTsuhp3igyeDVea6gSCdtrQUQDVHpW, address:145.79.227.2:4119): protocol: expected piece hash; context deadline exceeded; EOF
Apr 19 14:09:24.600415442 (286h23m11.419651496s) [noise=true , address=145.79.227.17:4972 , piece_id=57NAU5R6YCE4AA3PLAKII2U4K5KJCYDS3NH5PZAOUL36W7MSLONQ, seg_pos={0 0}, upload=274918]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: write tcp 188.166.225.170:55998->145.79.227.17:4972: use of closed network connection; write tcp 188.166.225.170:55998->145.79.227.17:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55998->145.79.227.17:4972: use of closed network connection
Apr 19 14:09:24.600668150 (286h23m11.419904204s) [noise=true , address=145.79.227.9:4366 , piece_id=D2KZ4TICJOQGO4FN5ITDMYVH6NH7FAFDKVNDYZTFSGDNGO3AT4FQ, seg_pos={0 0}, upload=274918]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: rpc: dial tcp 145.79.227.9:4366: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 19 14:09:24.600698107 (286h23m11.419934181s) [noise=true , address=145.79.227.36:4518 , piece_id=JC5WKYW543ZUQOAJG33PLM4A6A7WYGWBC6QLYCUKEGSSHYX6JDBA, seg_pos={0 0}, upload=274918]: Done uploading piece. err: ecclient: upload failed (node:17TZ9JLdCivFF46guhSTW6hcoUmTz5uo5wvLkDqC4BGmpm9eLb, address:145.79.227.36:4518): protocol: expected piece hash; context deadline exceeded; EOF
Apr 19 14:09:24.600963510 (286h23m11.420199584s) [noise=true , address=77.90.31.15:4700 , piece_id=CGWM3PZIGR2LX63KTTNJALEFI4NY2YVLBAZAXKGK2BE3V37D6GMA, seg_pos={0 0}, upload=274918]: Done uploading piece. err: ecclient: upload failed (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, address:77.90.31.15:4700): protocol: read tcp 188.166.225.170:48972->77.90.31.15:4700: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 19 14:09:24.601113419 (286h23m11.420349453s) [noise=true , address=145.79.227.34:4907 , piece_id=OVWJEETOWMYMW4OBXTEM2IR44YU2USTJHSKVONV4FMV2LOTFXM5Q, seg_pos={0 0}, upload=274918]: Done uploading piece. err: ecclient: upload failed (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS, address:145.79.227.34:4907): protocol: rpc: dial tcp 145.79.227.34:4907: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 19 14:09:24.601152354 (286h23m11.420388378s) [noise=true , address=45.91.116.7:4012 , piece_id=SDT4HR32F6WKFNFJXHMU4KMUGXUEJ32OYUPOC6SJBTATEXUL5RNQ, seg_pos={0 0}, upload=274918]: Done uploading piece. err: ecclient: upload failed (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, address:45.91.116.7:4012): protocol: read tcp 188.166.225.170:41998->45.91.116.7:4012: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 19 14:09:24.601398209 (286h23m11.420634233s) [noise=true , address=145.79.227.51:4415 , piece_id=RDAXQUREP4SXGB4TCQJUXD6YPRXXBFJIRFEPHUOA7ZYX6A5QOM2A, seg_pos={0 0}, upload=274918]: Done uploading piece. err: ecclient: upload failed (node:129sUhRR5RYtmrrfcEq2KgikZqZqQMRbwgK8NAKLFTTZgXDHW1v, address:145.79.227.51:4415): protocol: read tcp 188.166.225.170:59672->145.79.227.51:4415: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 19 14:09:24.601602834 (286h23m11.420838888s) [noise=true , address=145.79.227.37:4568 , piece_id=XT3EGREWAC2Q6RZXKJAM6ORKDYUSNRIKT5RJOMGEJ4B2N2PEC46A, seg_pos={0 0}, upload=274918]: Done uploading piece. err: ecclient: upload failed (node:1zJigHghNntzHNWim5RnWcMPfeUAbqsrvrfWretYcyEkXbSYzJ, address:145.79.227.37:4568): protocol: read tcp 188.166.225.170:49018->145.79.227.37:4568: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 19 14:09:24.601856011 (286h23m11.421092045s) [noise=true , address=145.79.227.13:4805 , piece_id=GGNMTIQPEMO7RN6KJFGCOUU6XX65CKHFJIZHFC2JVYR2JNJK3Y6Q, seg_pos={0 0}, upload=274918]: Done uploading piece. err: ecclient: upload failed (node:12YEVsEwZwfWsnZFyaSkd9Dn53Tb9EFUyWEVDwTQeMJkcjwrj7x, address:145.79.227.13:4805): protocol: read tcp 188.166.225.170:46898->145.79.227.13:4805: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 19 14:09:24.601963369 (286h23m11.421199393s) [noise=true , address=45.91.116.12:4227 , piece_id=LODLVJMI2EAMJ3EAAOI4Z4RWMOAQHFZL45MSWKVS5MB7FCNJ57AA, seg_pos={0 0}, upload=274918]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: read tcp 188.166.225.170:44102->45.91.116.12:4227: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 19 14:09:24.602008025 (286h23m11.42124405s) [seg_pos={0 0}, upload=274918]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil>
Apr 19 14:09:24.851854994 (286h23m11.671091018s) [upload=274918]: Done uploading segments...
Apr 19 14:09:24.851892576 (286h23m11.6711286s) [upload=274918]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5)
Duration: 5.763040751s
core
uplink
us-select-1
|
checkers.HTTP{"GET" "https://www.apple.com/robots.txt"} failure (antiblip window 3/60): (http.apple_robots_txt)antiblip:
1 out of 60 is failed (threshold=3) [..X.........................................................] duration:
15.219513ms remote:
"tcp:104.119.104.223:443" target:
https://www.apple.com/robots.txt trace:
https://jaeger.dev.storj.io/trace/505bb4ef80a6b4b1
FAIL (2026-04-19 14:51:15 UTC)
orbiter warning: Get "https://www.apple.com/robots.txt": EOF
Duration: 15.243068ms
core
website
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
1 out of 60 is failed (threshold=3) [....X.......................................................] duration:
5.606232892s trace:
https://jaeger.dev.storj.io/trace/59b3e23b0ad9afe7
FAIL (2026-04-19 14:51:05 UTC)
orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 39)
Duration: 5.606268812s
core
uplink
us1
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.remote)antiblip:
1 out of 60 is failed (threshold=3) [......X.....................................................] duration:
3.130302151s trace:
https://jaeger.dev.storj.io/trace/2fe350ab9c48d459
FAIL (2026-04-19 14:51:02 UTC)
orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 23)
Duration: 3.130321317s
core
uplink
ap1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..X.........................................................] duration:
5.729180787s trace:
https://jaeger.dev.storj.io/trace/2e2125521d7abc3e
FAIL (2026-04-19 14:50:49 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 5)
======LOGS======
e...
Apr 19 14:08:55.493690567 (286h22m42.312926591s) [noise=true , address=77.90.31.16:4738 , piece_id=NOF6IEE7PK5DHZ7FMWTRGL6WJ5I7D5XK7HEZKEUB7SCV6YGW3JQQ, seg_pos={0 0}, upload=274918]: Uploading piece...
Apr 19 14:08:55.494428361 (286h22m42.313664385s) [noise=true , address=145.79.227.13:4805 , piece_id=GGNMTIQPEMO7RN6KJFGCOUU6XX65CKHFJIZHFC2JVYR2JNJK3Y6Q, seg_pos={0 0}, upload=274918]: Uploading piece...
Apr 19 14:08:55.494992880 (286h22m42.314228904s) [noise=true , address=45.91.116.12:4227 , piece_id=LODLVJMI2EAMJ3EAAOI4Z4RWMOAQHFZL45MSWKVS5MB7FCNJ57AA, seg_pos={0 0}, upload=274918]: Uploading piece...
Apr 19 14:08:55.495362383 (286h22m42.314598407s) [noise=true , address=145.79.227.51:4415 , piece_id=RDAXQUREP4SXGB4TCQJUXD6YPRXXBFJIRFEPHUOA7ZYX6A5QOM2A, seg_pos={0 0}, upload=274918]: Uploading piece...
Apr 19 14:08:55.495930219 (286h22m42.315166243s) [noise=true , address=45.91.116.11:4152 , piece_id=O46QV2WY5ZRZSU3BJK32J6B2QAYIQH6VPSDB5LVZEHIL6FITZQQQ, seg_pos={0 0}, upload=274918]: Uploading piece...
Apr 19 14:08:55.496278582 (286h22m42.315514606s) [noise=true , address=145.79.227.15:4879 , piece_id=MWHLQQFFDK2FDKJN7J4MY7FY5O3MB6WMQFXBOLTXMRPRCTSVMQRQ, seg_pos={0 0}, upload=274918]: Uploading piece...
Apr 19 14:08:55.497181003 (286h22m42.316417047s) [noise=true , address=145.79.227.9:4366 , piece_id=D2KZ4TICJOQGO4FN5ITDMYVH6NH7FAFDKVNDYZTFSGDNGO3AT4FQ, seg_pos={0 0}, upload=274918]: Uploading piece...
Apr 19 14:08:55.497590223 (286h22m42.316826258s) [noise=true , address=145.79.227.34:4907 , piece_id=OVWJEETOWMYMW4OBXTEM2IR44YU2USTJHSKVONV4FMV2LOTFXM5Q, seg_pos={0 0}, upload=274918]: Uploading piece...
Apr 19 14:08:55.498297648 (286h22m42.317533672s) [noise=true , address=145.79.227.36:4518 , piece_id=JC5WKYW543ZUQOAJG33PLM4A6A7WYGWBC6QLYCUKEGSSHYX6JDBA, seg_pos={0 0}, upload=274918]: Uploading piece...
Apr 19 14:08:55.498945970 (286h22m42.318182014s) [noise=true , address=145.79.227.38:4982 , piece_id=ERRKLW4B4WI7YWVD2JJPEDHSU4IEJJM5UHFHTJ3CELIWXNZTJ34Q, seg_pos={0 0}, upload=274918]: Uploading piece...
Apr 19 14:08:55.499381660 (286h22m42.318617684s) [noise=true , address=145.79.227.2:4119 , piece_id=ME7PKNTEYNW36GBEP2LNSMUZMTS7JR67BEUHWEDYTWKB6ANYYHKQ, seg_pos={0 0}, upload=274918]: Uploading piece...
Apr 19 14:08:55.500079246 (286h22m42.31931527s) [noise=true , address=145.79.227.37:4568 , piece_id=XT3EGREWAC2Q6RZXKJAM6ORKDYUSNRIKT5RJOMGEJ4B2N2PEC46A, seg_pos={0 0}, upload=274918]: Uploading piece...
Apr 19 14:08:55.500330311 (286h22m42.319566335s) [noise=true , address=77.90.31.15:4700 , piece_id=CGWM3PZIGR2LX63KTTNJALEFI4NY2YVLBAZAXKGK2BE3V37D6GMA, seg_pos={0 0}, upload=274918]: Uploading piece...
Apr 19 14:08:55.501048167 (286h22m42.320284191s) [noise=true , address=145.79.227.3:4154 , piece_id=GCLCPG2QCIT4AMKHMAU4QZHPAONPDNSJAGLLVN4TO2VBZZGJOGIA, seg_pos={0 0}, upload=274918]: Uploading piece...
Apr 19 14:08:55.501263162 (286h22m42.320499186s) [noise=true , address=145.79.227.17:4972 , piece_id=57NAU5R6YCE4AA3PLAKII2U4K5KJCYDS3NH5PZAOUL36W7MSLONQ, seg_pos={0 0}, upload=274918]: Uploading piece...
Apr 19 14:08:57.949098467 (286h22m44.768334501s) [upload=274918]: Next returned nil segment
Apr 19 14:08:57.949115480 (286h22m44.768351554s) [upload=274918]: Waiting for error group managing segments...
Apr 19 14:08:58.345206270 (286h22m45.164442294s) [noise=true , address=145.79.227.3:4154 , piece_id=GCLCPG2QCIT4AMKHMAU4QZHPAONPDNSJAGLLVN4TO2VBZZGJOGIA, seg_pos={0 0}, upload=274918]: Done uploading piece. err: <nil>
Apr 19 14:09:01.082767579 (286h22m47.902003603s) [noise=true , address=45.91.116.11:4152 , piece_id=O46QV2WY5ZRZSU3BJK32J6B2QAYIQH6VPSDB5LVZEHIL6FITZQQQ, seg_pos={0 0}, upload=274918]: Done uploading piece. err: <nil>
Apr 19 14:09:05.145686115 (286h22m51.964922139s) [noise=true , address=77.90.31.16:4738 , piece_id=NOF6IEE7PK5DHZ7FMWTRGL6WJ5I7D5XK7HEZKEUB7SCV6YGW3JQQ, seg_pos={0 0}, upload=274918]: Done uploading piece. err: <nil>
Apr 19 14:09:10.080594511 (286h22m56.899830535s) [noise=true , address=145.79.227.38:4982 , piece_id=ERRKLW4B4WI7YWVD2JJPEDHSU4IEJJM5UHFHTJ3CELIWXNZTJ34Q, seg_pos={0 0}, upload=274918]: Done uploading piece. err: <nil>
Apr 19 14:09:11.441874037 (286h22m58.261110061s) [noise=true , address=145.79.227.15:4879 , piece_id=MWHLQQFFDK2FDKJN7J4MY7FY5O3MB6WMQFXBOLTXMRPRCTSVMQRQ, seg_pos={0 0}, upload=274918]: Done uploading piece. err: <nil>
Apr 19 14:09:24.600269009 (286h23m11.419505053s) [noise=true , address=145.79.227.2:4119 , piece_id=ME7PKNTEYNW36GBEP2LNSMUZMTS7JR67BEUHWEDYTWKB6ANYYHKQ, seg_pos={0 0}, upload=274918]: Done uploading piece. err: ecclient: upload failed (node:121xNAnkm9HZXUszp26UmWTsuhp3igyeDVea6gSCdtrQUQDVHpW, address:145.79.227.2:4119): protocol: expected piece hash; context deadline exceeded; EOF
Apr 19 14:09:24.600415442 (286h23m11.419651496s) [noise=true , address=145.79.227.17:4972 , piece_id=57NAU5R6YCE4AA3PLAKII2U4K5KJCYDS3NH5PZAOUL36W7MSLONQ, seg_pos={0 0}, upload=274918]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: write tcp 188.166.225.170:55998->145.79.227.17:4972: use of closed network connection; write tcp 188.166.225.170:55998->145.79.227.17:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55998->145.79.227.17:4972: use of closed network connection
Apr 19 14:09:24.600668150 (286h23m11.419904204s) [noise=true , address=145.79.227.9:4366 , piece_id=D2KZ4TICJOQGO4FN5ITDMYVH6NH7FAFDKVNDYZTFSGDNGO3AT4FQ, seg_pos={0 0}, upload=274918]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: rpc: dial tcp 145.79.227.9:4366: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 19 14:09:24.600698107 (286h23m11.419934181s) [noise=true , address=145.79.227.36:4518 , piece_id=JC5WKYW543ZUQOAJG33PLM4A6A7WYGWBC6QLYCUKEGSSHYX6JDBA, seg_pos={0 0}, upload=274918]: Done uploading piece. err: ecclient: upload failed (node:17TZ9JLdCivFF46guhSTW6hcoUmTz5uo5wvLkDqC4BGmpm9eLb, address:145.79.227.36:4518): protocol: expected piece hash; context deadline exceeded; EOF
Apr 19 14:09:24.600963510 (286h23m11.420199584s) [noise=true , address=77.90.31.15:4700 , piece_id=CGWM3PZIGR2LX63KTTNJALEFI4NY2YVLBAZAXKGK2BE3V37D6GMA, seg_pos={0 0}, upload=274918]: Done uploading piece. err: ecclient: upload failed (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, address:77.90.31.15:4700): protocol: read tcp 188.166.225.170:48972->77.90.31.15:4700: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 19 14:09:24.601113419 (286h23m11.420349453s) [noise=true , address=145.79.227.34:4907 , piece_id=OVWJEETOWMYMW4OBXTEM2IR44YU2USTJHSKVONV4FMV2LOTFXM5Q, seg_pos={0 0}, upload=274918]: Done uploading piece. err: ecclient: upload failed (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS, address:145.79.227.34:4907): protocol: rpc: dial tcp 145.79.227.34:4907: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 19 14:09:24.601152354 (286h23m11.420388378s) [noise=true , address=45.91.116.7:4012 , piece_id=SDT4HR32F6WKFNFJXHMU4KMUGXUEJ32OYUPOC6SJBTATEXUL5RNQ, seg_pos={0 0}, upload=274918]: Done uploading piece. err: ecclient: upload failed (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, address:45.91.116.7:4012): protocol: read tcp 188.166.225.170:41998->45.91.116.7:4012: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 19 14:09:24.601398209 (286h23m11.420634233s) [noise=true , address=145.79.227.51:4415 , piece_id=RDAXQUREP4SXGB4TCQJUXD6YPRXXBFJIRFEPHUOA7ZYX6A5QOM2A, seg_pos={0 0}, upload=274918]: Done uploading piece. err: ecclient: upload failed (node:129sUhRR5RYtmrrfcEq2KgikZqZqQMRbwgK8NAKLFTTZgXDHW1v, address:145.79.227.51:4415): protocol: read tcp 188.166.225.170:59672->145.79.227.51:4415: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 19 14:09:24.601602834 (286h23m11.420838888s) [noise=true , address=145.79.227.37:4568 , piece_id=XT3EGREWAC2Q6RZXKJAM6ORKDYUSNRIKT5RJOMGEJ4B2N2PEC46A, seg_pos={0 0}, upload=274918]: Done uploading piece. err: ecclient: upload failed (node:1zJigHghNntzHNWim5RnWcMPfeUAbqsrvrfWretYcyEkXbSYzJ, address:145.79.227.37:4568): protocol: read tcp 188.166.225.170:49018->145.79.227.37:4568: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 19 14:09:24.601856011 (286h23m11.421092045s) [noise=true , address=145.79.227.13:4805 , piece_id=GGNMTIQPEMO7RN6KJFGCOUU6XX65CKHFJIZHFC2JVYR2JNJK3Y6Q, seg_pos={0 0}, upload=274918]: Done uploading piece. err: ecclient: upload failed (node:12YEVsEwZwfWsnZFyaSkd9Dn53Tb9EFUyWEVDwTQeMJkcjwrj7x, address:145.79.227.13:4805): protocol: read tcp 188.166.225.170:46898->145.79.227.13:4805: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 19 14:09:24.601963369 (286h23m11.421199393s) [noise=true , address=45.91.116.12:4227 , piece_id=LODLVJMI2EAMJ3EAAOI4Z4RWMOAQHFZL45MSWKVS5MB7FCNJ57AA, seg_pos={0 0}, upload=274918]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: read tcp 188.166.225.170:44102->45.91.116.12:4227: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 19 14:09:24.602008025 (286h23m11.42124405s) [seg_pos={0 0}, upload=274918]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil>
Apr 19 14:09:24.851854994 (286h23m11.671091018s) [upload=274918]: Done uploading segments...
Apr 19 14:09:24.851892576 (286h23m11.6711286s) [upload=274918]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5)
Duration: 5.729212449s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [........X...................................................] duration:
3.506574542s trace:
https://jaeger.dev.storj.io/trace/3c23449320be8b26
FAIL (2026-04-19 14:50:30 UTC)
orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded
Duration: 3.506602396s
core
uplink
us-select-1
us1-placement-20
|
checkers.HTTP{"GET" "https://www.apple.com/robots.txt"} failure (antiblip window 3/60): (http.apple_robots_txt)antiblip:
1 out of 60 is failed (threshold=3) [...X........................................................] duration:
15.158113ms remote:
"tcp:104.119.104.223:443" target:
https://www.apple.com/robots.txt trace:
https://jaeger.dev.storj.io/trace/505bb4ef80a6b4b1
FAIL (2026-04-19 14:50:26 UTC)
orbiter warning: Get "https://www.apple.com/robots.txt": EOF
Duration: 15.18192ms
core
website
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.......X....................................................] duration:
3.094628933s trace:
https://jaeger.dev.storj.io/trace/2fe350ab9c48d459
FAIL (2026-04-19 14:50:14 UTC)
orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 23)
Duration: 3.09465791s
core
uplink
ap1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [.........X..................................................] duration:
3.457892539s trace:
https://jaeger.dev.storj.io/trace/3c23449320be8b26
FAIL (2026-04-19 14:50:09 UTC)
orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded
Duration: 3.457937716s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.....X......................................................] duration:
5.556161107s trace:
https://jaeger.dev.storj.io/trace/59b3e23b0ad9afe7
FAIL (2026-04-19 14:50:08 UTC)
orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 39)
Duration: 5.556193158s
core
uplink
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...X........................................................] duration:
5.734125051s trace:
https://jaeger.dev.storj.io/trace/2e2125521d7abc3e
FAIL (2026-04-19 14:49:58 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 5)
======LOGS======
e...
Apr 19 14:08:55.493690567 (286h22m42.312926591s) [noise=true , address=77.90.31.16:4738 , piece_id=NOF6IEE7PK5DHZ7FMWTRGL6WJ5I7D5XK7HEZKEUB7SCV6YGW3JQQ, seg_pos={0 0}, upload=274918]: Uploading piece...
Apr 19 14:08:55.494428361 (286h22m42.313664385s) [noise=true , address=145.79.227.13:4805 , piece_id=GGNMTIQPEMO7RN6KJFGCOUU6XX65CKHFJIZHFC2JVYR2JNJK3Y6Q, seg_pos={0 0}, upload=274918]: Uploading piece...
Apr 19 14:08:55.494992880 (286h22m42.314228904s) [noise=true , address=45.91.116.12:4227 , piece_id=LODLVJMI2EAMJ3EAAOI4Z4RWMOAQHFZL45MSWKVS5MB7FCNJ57AA, seg_pos={0 0}, upload=274918]: Uploading piece...
Apr 19 14:08:55.495362383 (286h22m42.314598407s) [noise=true , address=145.79.227.51:4415 , piece_id=RDAXQUREP4SXGB4TCQJUXD6YPRXXBFJIRFEPHUOA7ZYX6A5QOM2A, seg_pos={0 0}, upload=274918]: Uploading piece...
Apr 19 14:08:55.495930219 (286h22m42.315166243s) [noise=true , address=45.91.116.11:4152 , piece_id=O46QV2WY5ZRZSU3BJK32J6B2QAYIQH6VPSDB5LVZEHIL6FITZQQQ, seg_pos={0 0}, upload=274918]: Uploading piece...
Apr 19 14:08:55.496278582 (286h22m42.315514606s) [noise=true , address=145.79.227.15:4879 , piece_id=MWHLQQFFDK2FDKJN7J4MY7FY5O3MB6WMQFXBOLTXMRPRCTSVMQRQ, seg_pos={0 0}, upload=274918]: Uploading piece...
Apr 19 14:08:55.497181003 (286h22m42.316417047s) [noise=true , address=145.79.227.9:4366 , piece_id=D2KZ4TICJOQGO4FN5ITDMYVH6NH7FAFDKVNDYZTFSGDNGO3AT4FQ, seg_pos={0 0}, upload=274918]: Uploading piece...
Apr 19 14:08:55.497590223 (286h22m42.316826258s) [noise=true , address=145.79.227.34:4907 , piece_id=OVWJEETOWMYMW4OBXTEM2IR44YU2USTJHSKVONV4FMV2LOTFXM5Q, seg_pos={0 0}, upload=274918]: Uploading piece...
Apr 19 14:08:55.498297648 (286h22m42.317533672s) [noise=true , address=145.79.227.36:4518 , piece_id=JC5WKYW543ZUQOAJG33PLM4A6A7WYGWBC6QLYCUKEGSSHYX6JDBA, seg_pos={0 0}, upload=274918]: Uploading piece...
Apr 19 14:08:55.498945970 (286h22m42.318182014s) [noise=true , address=145.79.227.38:4982 , piece_id=ERRKLW4B4WI7YWVD2JJPEDHSU4IEJJM5UHFHTJ3CELIWXNZTJ34Q, seg_pos={0 0}, upload=274918]: Uploading piece...
Apr 19 14:08:55.499381660 (286h22m42.318617684s) [noise=true , address=145.79.227.2:4119 , piece_id=ME7PKNTEYNW36GBEP2LNSMUZMTS7JR67BEUHWEDYTWKB6ANYYHKQ, seg_pos={0 0}, upload=274918]: Uploading piece...
Apr 19 14:08:55.500079246 (286h22m42.31931527s) [noise=true , address=145.79.227.37:4568 , piece_id=XT3EGREWAC2Q6RZXKJAM6ORKDYUSNRIKT5RJOMGEJ4B2N2PEC46A, seg_pos={0 0}, upload=274918]: Uploading piece...
Apr 19 14:08:55.500330311 (286h22m42.319566335s) [noise=true , address=77.90.31.15:4700 , piece_id=CGWM3PZIGR2LX63KTTNJALEFI4NY2YVLBAZAXKGK2BE3V37D6GMA, seg_pos={0 0}, upload=274918]: Uploading piece...
Apr 19 14:08:55.501048167 (286h22m42.320284191s) [noise=true , address=145.79.227.3:4154 , piece_id=GCLCPG2QCIT4AMKHMAU4QZHPAONPDNSJAGLLVN4TO2VBZZGJOGIA, seg_pos={0 0}, upload=274918]: Uploading piece...
Apr 19 14:08:55.501263162 (286h22m42.320499186s) [noise=true , address=145.79.227.17:4972 , piece_id=57NAU5R6YCE4AA3PLAKII2U4K5KJCYDS3NH5PZAOUL36W7MSLONQ, seg_pos={0 0}, upload=274918]: Uploading piece...
Apr 19 14:08:57.949098467 (286h22m44.768334501s) [upload=274918]: Next returned nil segment
Apr 19 14:08:57.949115480 (286h22m44.768351554s) [upload=274918]: Waiting for error group managing segments...
Apr 19 14:08:58.345206270 (286h22m45.164442294s) [noise=true , address=145.79.227.3:4154 , piece_id=GCLCPG2QCIT4AMKHMAU4QZHPAONPDNSJAGLLVN4TO2VBZZGJOGIA, seg_pos={0 0}, upload=274918]: Done uploading piece. err: <nil>
Apr 19 14:09:01.082767579 (286h22m47.902003603s) [noise=true , address=45.91.116.11:4152 , piece_id=O46QV2WY5ZRZSU3BJK32J6B2QAYIQH6VPSDB5LVZEHIL6FITZQQQ, seg_pos={0 0}, upload=274918]: Done uploading piece. err: <nil>
Apr 19 14:09:05.145686115 (286h22m51.964922139s) [noise=true , address=77.90.31.16:4738 , piece_id=NOF6IEE7PK5DHZ7FMWTRGL6WJ5I7D5XK7HEZKEUB7SCV6YGW3JQQ, seg_pos={0 0}, upload=274918]: Done uploading piece. err: <nil>
Apr 19 14:09:10.080594511 (286h22m56.899830535s) [noise=true , address=145.79.227.38:4982 , piece_id=ERRKLW4B4WI7YWVD2JJPEDHSU4IEJJM5UHFHTJ3CELIWXNZTJ34Q, seg_pos={0 0}, upload=274918]: Done uploading piece. err: <nil>
Apr 19 14:09:11.441874037 (286h22m58.261110061s) [noise=true , address=145.79.227.15:4879 , piece_id=MWHLQQFFDK2FDKJN7J4MY7FY5O3MB6WMQFXBOLTXMRPRCTSVMQRQ, seg_pos={0 0}, upload=274918]: Done uploading piece. err: <nil>
Apr 19 14:09:24.600269009 (286h23m11.419505053s) [noise=true , address=145.79.227.2:4119 , piece_id=ME7PKNTEYNW36GBEP2LNSMUZMTS7JR67BEUHWEDYTWKB6ANYYHKQ, seg_pos={0 0}, upload=274918]: Done uploading piece. err: ecclient: upload failed (node:121xNAnkm9HZXUszp26UmWTsuhp3igyeDVea6gSCdtrQUQDVHpW, address:145.79.227.2:4119): protocol: expected piece hash; context deadline exceeded; EOF
Apr 19 14:09:24.600415442 (286h23m11.419651496s) [noise=true , address=145.79.227.17:4972 , piece_id=57NAU5R6YCE4AA3PLAKII2U4K5KJCYDS3NH5PZAOUL36W7MSLONQ, seg_pos={0 0}, upload=274918]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: write tcp 188.166.225.170:55998->145.79.227.17:4972: use of closed network connection; write tcp 188.166.225.170:55998->145.79.227.17:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55998->145.79.227.17:4972: use of closed network connection
Apr 19 14:09:24.600668150 (286h23m11.419904204s) [noise=true , address=145.79.227.9:4366 , piece_id=D2KZ4TICJOQGO4FN5ITDMYVH6NH7FAFDKVNDYZTFSGDNGO3AT4FQ, seg_pos={0 0}, upload=274918]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: rpc: dial tcp 145.79.227.9:4366: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 19 14:09:24.600698107 (286h23m11.419934181s) [noise=true , address=145.79.227.36:4518 , piece_id=JC5WKYW543ZUQOAJG33PLM4A6A7WYGWBC6QLYCUKEGSSHYX6JDBA, seg_pos={0 0}, upload=274918]: Done uploading piece. err: ecclient: upload failed (node:17TZ9JLdCivFF46guhSTW6hcoUmTz5uo5wvLkDqC4BGmpm9eLb, address:145.79.227.36:4518): protocol: expected piece hash; context deadline exceeded; EOF
Apr 19 14:09:24.600963510 (286h23m11.420199584s) [noise=true , address=77.90.31.15:4700 , piece_id=CGWM3PZIGR2LX63KTTNJALEFI4NY2YVLBAZAXKGK2BE3V37D6GMA, seg_pos={0 0}, upload=274918]: Done uploading piece. err: ecclient: upload failed (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, address:77.90.31.15:4700): protocol: read tcp 188.166.225.170:48972->77.90.31.15:4700: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 19 14:09:24.601113419 (286h23m11.420349453s) [noise=true , address=145.79.227.34:4907 , piece_id=OVWJEETOWMYMW4OBXTEM2IR44YU2USTJHSKVONV4FMV2LOTFXM5Q, seg_pos={0 0}, upload=274918]: Done uploading piece. err: ecclient: upload failed (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS, address:145.79.227.34:4907): protocol: rpc: dial tcp 145.79.227.34:4907: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 19 14:09:24.601152354 (286h23m11.420388378s) [noise=true , address=45.91.116.7:4012 , piece_id=SDT4HR32F6WKFNFJXHMU4KMUGXUEJ32OYUPOC6SJBTATEXUL5RNQ, seg_pos={0 0}, upload=274918]: Done uploading piece. err: ecclient: upload failed (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, address:45.91.116.7:4012): protocol: read tcp 188.166.225.170:41998->45.91.116.7:4012: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 19 14:09:24.601398209 (286h23m11.420634233s) [noise=true , address=145.79.227.51:4415 , piece_id=RDAXQUREP4SXGB4TCQJUXD6YPRXXBFJIRFEPHUOA7ZYX6A5QOM2A, seg_pos={0 0}, upload=274918]: Done uploading piece. err: ecclient: upload failed (node:129sUhRR5RYtmrrfcEq2KgikZqZqQMRbwgK8NAKLFTTZgXDHW1v, address:145.79.227.51:4415): protocol: read tcp 188.166.225.170:59672->145.79.227.51:4415: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 19 14:09:24.601602834 (286h23m11.420838888s) [noise=true , address=145.79.227.37:4568 , piece_id=XT3EGREWAC2Q6RZXKJAM6ORKDYUSNRIKT5RJOMGEJ4B2N2PEC46A, seg_pos={0 0}, upload=274918]: Done uploading piece. err: ecclient: upload failed (node:1zJigHghNntzHNWim5RnWcMPfeUAbqsrvrfWretYcyEkXbSYzJ, address:145.79.227.37:4568): protocol: read tcp 188.166.225.170:49018->145.79.227.37:4568: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 19 14:09:24.601856011 (286h23m11.421092045s) [noise=true , address=145.79.227.13:4805 , piece_id=GGNMTIQPEMO7RN6KJFGCOUU6XX65CKHFJIZHFC2JVYR2JNJK3Y6Q, seg_pos={0 0}, upload=274918]: Done uploading piece. err: ecclient: upload failed (node:12YEVsEwZwfWsnZFyaSkd9Dn53Tb9EFUyWEVDwTQeMJkcjwrj7x, address:145.79.227.13:4805): protocol: read tcp 188.166.225.170:46898->145.79.227.13:4805: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 19 14:09:24.601963369 (286h23m11.421199393s) [noise=true , address=45.91.116.12:4227 , piece_id=LODLVJMI2EAMJ3EAAOI4Z4RWMOAQHFZL45MSWKVS5MB7FCNJ57AA, seg_pos={0 0}, upload=274918]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: read tcp 188.166.225.170:44102->45.91.116.12:4227: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 19 14:09:24.602008025 (286h23m11.42124405s) [seg_pos={0 0}, upload=274918]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil>
Apr 19 14:09:24.851854994 (286h23m11.671091018s) [upload=274918]: Done uploading segments...
Apr 19 14:09:24.851892576 (286h23m11.6711286s) [upload=274918]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5)
Duration: 5.734149798s
core
uplink
us-select-1
|
checkers.HTTP{"GET" "https://www.apple.com/robots.txt"} failure (antiblip window 3/60): (http.apple_robots_txt)antiblip:
1 out of 60 is failed (threshold=3) [....X.......................................................] duration:
12.211798ms remote:
"tcp:104.119.104.223:443" target:
https://www.apple.com/robots.txt trace:
https://jaeger.dev.storj.io/trace/505bb4ef80a6b4b1
FAIL (2026-04-19 14:49:57 UTC)
orbiter warning: Get "https://www.apple.com/robots.txt": EOF
Duration: 12.231356ms
core
website
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [....X.......................................................] duration:
5.673118538s trace:
https://jaeger.dev.storj.io/trace/2e2125521d7abc3e
FAIL (2026-04-19 14:49:38 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 5)
======LOGS======
e...
Apr 19 14:08:55.493690567 (286h22m42.312926591s) [noise=true , address=77.90.31.16:4738 , piece_id=NOF6IEE7PK5DHZ7FMWTRGL6WJ5I7D5XK7HEZKEUB7SCV6YGW3JQQ, seg_pos={0 0}, upload=274918]: Uploading piece...
Apr 19 14:08:55.494428361 (286h22m42.313664385s) [noise=true , address=145.79.227.13:4805 , piece_id=GGNMTIQPEMO7RN6KJFGCOUU6XX65CKHFJIZHFC2JVYR2JNJK3Y6Q, seg_pos={0 0}, upload=274918]: Uploading piece...
Apr 19 14:08:55.494992880 (286h22m42.314228904s) [noise=true , address=45.91.116.12:4227 , piece_id=LODLVJMI2EAMJ3EAAOI4Z4RWMOAQHFZL45MSWKVS5MB7FCNJ57AA, seg_pos={0 0}, upload=274918]: Uploading piece...
Apr 19 14:08:55.495362383 (286h22m42.314598407s) [noise=true , address=145.79.227.51:4415 , piece_id=RDAXQUREP4SXGB4TCQJUXD6YPRXXBFJIRFEPHUOA7ZYX6A5QOM2A, seg_pos={0 0}, upload=274918]: Uploading piece...
Apr 19 14:08:55.495930219 (286h22m42.315166243s) [noise=true , address=45.91.116.11:4152 , piece_id=O46QV2WY5ZRZSU3BJK32J6B2QAYIQH6VPSDB5LVZEHIL6FITZQQQ, seg_pos={0 0}, upload=274918]: Uploading piece...
Apr 19 14:08:55.496278582 (286h22m42.315514606s) [noise=true , address=145.79.227.15:4879 , piece_id=MWHLQQFFDK2FDKJN7J4MY7FY5O3MB6WMQFXBOLTXMRPRCTSVMQRQ, seg_pos={0 0}, upload=274918]: Uploading piece...
Apr 19 14:08:55.497181003 (286h22m42.316417047s) [noise=true , address=145.79.227.9:4366 , piece_id=D2KZ4TICJOQGO4FN5ITDMYVH6NH7FAFDKVNDYZTFSGDNGO3AT4FQ, seg_pos={0 0}, upload=274918]: Uploading piece...
Apr 19 14:08:55.497590223 (286h22m42.316826258s) [noise=true , address=145.79.227.34:4907 , piece_id=OVWJEETOWMYMW4OBXTEM2IR44YU2USTJHSKVONV4FMV2LOTFXM5Q, seg_pos={0 0}, upload=274918]: Uploading piece...
Apr 19 14:08:55.498297648 (286h22m42.317533672s) [noise=true , address=145.79.227.36:4518 , piece_id=JC5WKYW543ZUQOAJG33PLM4A6A7WYGWBC6QLYCUKEGSSHYX6JDBA, seg_pos={0 0}, upload=274918]: Uploading piece...
Apr 19 14:08:55.498945970 (286h22m42.318182014s) [noise=true , address=145.79.227.38:4982 , piece_id=ERRKLW4B4WI7YWVD2JJPEDHSU4IEJJM5UHFHTJ3CELIWXNZTJ34Q, seg_pos={0 0}, upload=274918]: Uploading piece...
Apr 19 14:08:55.499381660 (286h22m42.318617684s) [noise=true , address=145.79.227.2:4119 , piece_id=ME7PKNTEYNW36GBEP2LNSMUZMTS7JR67BEUHWEDYTWKB6ANYYHKQ, seg_pos={0 0}, upload=274918]: Uploading piece...
Apr 19 14:08:55.500079246 (286h22m42.31931527s) [noise=true , address=145.79.227.37:4568 , piece_id=XT3EGREWAC2Q6RZXKJAM6ORKDYUSNRIKT5RJOMGEJ4B2N2PEC46A, seg_pos={0 0}, upload=274918]: Uploading piece...
Apr 19 14:08:55.500330311 (286h22m42.319566335s) [noise=true , address=77.90.31.15:4700 , piece_id=CGWM3PZIGR2LX63KTTNJALEFI4NY2YVLBAZAXKGK2BE3V37D6GMA, seg_pos={0 0}, upload=274918]: Uploading piece...
Apr 19 14:08:55.501048167 (286h22m42.320284191s) [noise=true , address=145.79.227.3:4154 , piece_id=GCLCPG2QCIT4AMKHMAU4QZHPAONPDNSJAGLLVN4TO2VBZZGJOGIA, seg_pos={0 0}, upload=274918]: Uploading piece...
Apr 19 14:08:55.501263162 (286h22m42.320499186s) [noise=true , address=145.79.227.17:4972 , piece_id=57NAU5R6YCE4AA3PLAKII2U4K5KJCYDS3NH5PZAOUL36W7MSLONQ, seg_pos={0 0}, upload=274918]: Uploading piece...
Apr 19 14:08:57.949098467 (286h22m44.768334501s) [upload=274918]: Next returned nil segment
Apr 19 14:08:57.949115480 (286h22m44.768351554s) [upload=274918]: Waiting for error group managing segments...
Apr 19 14:08:58.345206270 (286h22m45.164442294s) [noise=true , address=145.79.227.3:4154 , piece_id=GCLCPG2QCIT4AMKHMAU4QZHPAONPDNSJAGLLVN4TO2VBZZGJOGIA, seg_pos={0 0}, upload=274918]: Done uploading piece. err: <nil>
Apr 19 14:09:01.082767579 (286h22m47.902003603s) [noise=true , address=45.91.116.11:4152 , piece_id=O46QV2WY5ZRZSU3BJK32J6B2QAYIQH6VPSDB5LVZEHIL6FITZQQQ, seg_pos={0 0}, upload=274918]: Done uploading piece. err: <nil>
Apr 19 14:09:05.145686115 (286h22m51.964922139s) [noise=true , address=77.90.31.16:4738 , piece_id=NOF6IEE7PK5DHZ7FMWTRGL6WJ5I7D5XK7HEZKEUB7SCV6YGW3JQQ, seg_pos={0 0}, upload=274918]: Done uploading piece. err: <nil>
Apr 19 14:09:10.080594511 (286h22m56.899830535s) [noise=true , address=145.79.227.38:4982 , piece_id=ERRKLW4B4WI7YWVD2JJPEDHSU4IEJJM5UHFHTJ3CELIWXNZTJ34Q, seg_pos={0 0}, upload=274918]: Done uploading piece. err: <nil>
Apr 19 14:09:11.441874037 (286h22m58.261110061s) [noise=true , address=145.79.227.15:4879 , piece_id=MWHLQQFFDK2FDKJN7J4MY7FY5O3MB6WMQFXBOLTXMRPRCTSVMQRQ, seg_pos={0 0}, upload=274918]: Done uploading piece. err: <nil>
Apr 19 14:09:24.600269009 (286h23m11.419505053s) [noise=true , address=145.79.227.2:4119 , piece_id=ME7PKNTEYNW36GBEP2LNSMUZMTS7JR67BEUHWEDYTWKB6ANYYHKQ, seg_pos={0 0}, upload=274918]: Done uploading piece. err: ecclient: upload failed (node:121xNAnkm9HZXUszp26UmWTsuhp3igyeDVea6gSCdtrQUQDVHpW, address:145.79.227.2:4119): protocol: expected piece hash; context deadline exceeded; EOF
Apr 19 14:09:24.600415442 (286h23m11.419651496s) [noise=true , address=145.79.227.17:4972 , piece_id=57NAU5R6YCE4AA3PLAKII2U4K5KJCYDS3NH5PZAOUL36W7MSLONQ, seg_pos={0 0}, upload=274918]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: write tcp 188.166.225.170:55998->145.79.227.17:4972: use of closed network connection; write tcp 188.166.225.170:55998->145.79.227.17:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55998->145.79.227.17:4972: use of closed network connection
Apr 19 14:09:24.600668150 (286h23m11.419904204s) [noise=true , address=145.79.227.9:4366 , piece_id=D2KZ4TICJOQGO4FN5ITDMYVH6NH7FAFDKVNDYZTFSGDNGO3AT4FQ, seg_pos={0 0}, upload=274918]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: rpc: dial tcp 145.79.227.9:4366: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 19 14:09:24.600698107 (286h23m11.419934181s) [noise=true , address=145.79.227.36:4518 , piece_id=JC5WKYW543ZUQOAJG33PLM4A6A7WYGWBC6QLYCUKEGSSHYX6JDBA, seg_pos={0 0}, upload=274918]: Done uploading piece. err: ecclient: upload failed (node:17TZ9JLdCivFF46guhSTW6hcoUmTz5uo5wvLkDqC4BGmpm9eLb, address:145.79.227.36:4518): protocol: expected piece hash; context deadline exceeded; EOF
Apr 19 14:09:24.600963510 (286h23m11.420199584s) [noise=true , address=77.90.31.15:4700 , piece_id=CGWM3PZIGR2LX63KTTNJALEFI4NY2YVLBAZAXKGK2BE3V37D6GMA, seg_pos={0 0}, upload=274918]: Done uploading piece. err: ecclient: upload failed (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, address:77.90.31.15:4700): protocol: read tcp 188.166.225.170:48972->77.90.31.15:4700: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 19 14:09:24.601113419 (286h23m11.420349453s) [noise=true , address=145.79.227.34:4907 , piece_id=OVWJEETOWMYMW4OBXTEM2IR44YU2USTJHSKVONV4FMV2LOTFXM5Q, seg_pos={0 0}, upload=274918]: Done uploading piece. err: ecclient: upload failed (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS, address:145.79.227.34:4907): protocol: rpc: dial tcp 145.79.227.34:4907: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 19 14:09:24.601152354 (286h23m11.420388378s) [noise=true , address=45.91.116.7:4012 , piece_id=SDT4HR32F6WKFNFJXHMU4KMUGXUEJ32OYUPOC6SJBTATEXUL5RNQ, seg_pos={0 0}, upload=274918]: Done uploading piece. err: ecclient: upload failed (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, address:45.91.116.7:4012): protocol: read tcp 188.166.225.170:41998->45.91.116.7:4012: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 19 14:09:24.601398209 (286h23m11.420634233s) [noise=true , address=145.79.227.51:4415 , piece_id=RDAXQUREP4SXGB4TCQJUXD6YPRXXBFJIRFEPHUOA7ZYX6A5QOM2A, seg_pos={0 0}, upload=274918]: Done uploading piece. err: ecclient: upload failed (node:129sUhRR5RYtmrrfcEq2KgikZqZqQMRbwgK8NAKLFTTZgXDHW1v, address:145.79.227.51:4415): protocol: read tcp 188.166.225.170:59672->145.79.227.51:4415: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 19 14:09:24.601602834 (286h23m11.420838888s) [noise=true , address=145.79.227.37:4568 , piece_id=XT3EGREWAC2Q6RZXKJAM6ORKDYUSNRIKT5RJOMGEJ4B2N2PEC46A, seg_pos={0 0}, upload=274918]: Done uploading piece. err: ecclient: upload failed (node:1zJigHghNntzHNWim5RnWcMPfeUAbqsrvrfWretYcyEkXbSYzJ, address:145.79.227.37:4568): protocol: read tcp 188.166.225.170:49018->145.79.227.37:4568: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 19 14:09:24.601856011 (286h23m11.421092045s) [noise=true , address=145.79.227.13:4805 , piece_id=GGNMTIQPEMO7RN6KJFGCOUU6XX65CKHFJIZHFC2JVYR2JNJK3Y6Q, seg_pos={0 0}, upload=274918]: Done uploading piece. err: ecclient: upload failed (node:12YEVsEwZwfWsnZFyaSkd9Dn53Tb9EFUyWEVDwTQeMJkcjwrj7x, address:145.79.227.13:4805): protocol: read tcp 188.166.225.170:46898->145.79.227.13:4805: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 19 14:09:24.601963369 (286h23m11.421199393s) [noise=true , address=45.91.116.12:4227 , piece_id=LODLVJMI2EAMJ3EAAOI4Z4RWMOAQHFZL45MSWKVS5MB7FCNJ57AA, seg_pos={0 0}, upload=274918]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: read tcp 188.166.225.170:44102->45.91.116.12:4227: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 19 14:09:24.602008025 (286h23m11.42124405s) [seg_pos={0 0}, upload=274918]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil>
Apr 19 14:09:24.851854994 (286h23m11.671091018s) [upload=274918]: Done uploading segments...
Apr 19 14:09:24.851892576 (286h23m11.6711286s) [upload=274918]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5)
Duration: 5.673145251s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
1 out of 60 is failed (threshold=3) [......X.....................................................] duration:
5.403087045s trace:
https://jaeger.dev.storj.io/trace/59b3e23b0ad9afe7
FAIL (2026-04-19 14:49:29 UTC)
orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 39)
Duration: 5.403115741s
core
uplink
us1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [..........X.................................................] duration:
3.45351329s trace:
https://jaeger.dev.storj.io/trace/3c23449320be8b26
FAIL (2026-04-19 14:49:26 UTC)
orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded
Duration: 3.45355969s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.remote)antiblip:
1 out of 60 is failed (threshold=3) [........X...................................................] duration:
3.051509807s trace:
https://jaeger.dev.storj.io/trace/2fe350ab9c48d459
FAIL (2026-04-19 14:49:17 UTC)
orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 23)
Duration: 3.051539074s
core
uplink
ap1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [...........X................................................] duration:
3.3793729s trace:
https://jaeger.dev.storj.io/trace/3c23449320be8b26
FAIL (2026-04-19 14:49:10 UTC)
orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded
Duration: 3.379402899s
core
uplink
us-select-1
us1-placement-20
|
checkers.HTTP{"GET" "https://www.apple.com/robots.txt"} failure (antiblip window 3/60): (http.apple_robots_txt)antiblip:
1 out of 60 is failed (threshold=3) [.....X......................................................] duration:
13.110292ms remote:
"tcp:104.119.104.223:443" target:
https://www.apple.com/robots.txt trace:
https://jaeger.dev.storj.io/trace/505bb4ef80a6b4b1
FAIL (2026-04-19 14:49:05 UTC)
orbiter warning: Get "https://www.apple.com/robots.txt": EOF
Duration: 13.135932ms
core
website
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.....X......................................................] duration:
5.750562252s trace:
https://jaeger.dev.storj.io/trace/2e2125521d7abc3e
FAIL (2026-04-19 14:48:49 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 5)
======LOGS======
e...
Apr 19 14:08:55.493690567 (286h22m42.312926591s) [noise=true , address=77.90.31.16:4738 , piece_id=NOF6IEE7PK5DHZ7FMWTRGL6WJ5I7D5XK7HEZKEUB7SCV6YGW3JQQ, seg_pos={0 0}, upload=274918]: Uploading piece...
Apr 19 14:08:55.494428361 (286h22m42.313664385s) [noise=true , address=145.79.227.13:4805 , piece_id=GGNMTIQPEMO7RN6KJFGCOUU6XX65CKHFJIZHFC2JVYR2JNJK3Y6Q, seg_pos={0 0}, upload=274918]: Uploading piece...
Apr 19 14:08:55.494992880 (286h22m42.314228904s) [noise=true , address=45.91.116.12:4227 , piece_id=LODLVJMI2EAMJ3EAAOI4Z4RWMOAQHFZL45MSWKVS5MB7FCNJ57AA, seg_pos={0 0}, upload=274918]: Uploading piece...
Apr 19 14:08:55.495362383 (286h22m42.314598407s) [noise=true , address=145.79.227.51:4415 , piece_id=RDAXQUREP4SXGB4TCQJUXD6YPRXXBFJIRFEPHUOA7ZYX6A5QOM2A, seg_pos={0 0}, upload=274918]: Uploading piece...
Apr 19 14:08:55.495930219 (286h22m42.315166243s) [noise=true , address=45.91.116.11:4152 , piece_id=O46QV2WY5ZRZSU3BJK32J6B2QAYIQH6VPSDB5LVZEHIL6FITZQQQ, seg_pos={0 0}, upload=274918]: Uploading piece...
Apr 19 14:08:55.496278582 (286h22m42.315514606s) [noise=true , address=145.79.227.15:4879 , piece_id=MWHLQQFFDK2FDKJN7J4MY7FY5O3MB6WMQFXBOLTXMRPRCTSVMQRQ, seg_pos={0 0}, upload=274918]: Uploading piece...
Apr 19 14:08:55.497181003 (286h22m42.316417047s) [noise=true , address=145.79.227.9:4366 , piece_id=D2KZ4TICJOQGO4FN5ITDMYVH6NH7FAFDKVNDYZTFSGDNGO3AT4FQ, seg_pos={0 0}, upload=274918]: Uploading piece...
Apr 19 14:08:55.497590223 (286h22m42.316826258s) [noise=true , address=145.79.227.34:4907 , piece_id=OVWJEETOWMYMW4OBXTEM2IR44YU2USTJHSKVONV4FMV2LOTFXM5Q, seg_pos={0 0}, upload=274918]: Uploading piece...
Apr 19 14:08:55.498297648 (286h22m42.317533672s) [noise=true , address=145.79.227.36:4518 , piece_id=JC5WKYW543ZUQOAJG33PLM4A6A7WYGWBC6QLYCUKEGSSHYX6JDBA, seg_pos={0 0}, upload=274918]: Uploading piece...
Apr 19 14:08:55.498945970 (286h22m42.318182014s) [noise=true , address=145.79.227.38:4982 , piece_id=ERRKLW4B4WI7YWVD2JJPEDHSU4IEJJM5UHFHTJ3CELIWXNZTJ34Q, seg_pos={0 0}, upload=274918]: Uploading piece...
Apr 19 14:08:55.499381660 (286h22m42.318617684s) [noise=true , address=145.79.227.2:4119 , piece_id=ME7PKNTEYNW36GBEP2LNSMUZMTS7JR67BEUHWEDYTWKB6ANYYHKQ, seg_pos={0 0}, upload=274918]: Uploading piece...
Apr 19 14:08:55.500079246 (286h22m42.31931527s) [noise=true , address=145.79.227.37:4568 , piece_id=XT3EGREWAC2Q6RZXKJAM6ORKDYUSNRIKT5RJOMGEJ4B2N2PEC46A, seg_pos={0 0}, upload=274918]: Uploading piece...
Apr 19 14:08:55.500330311 (286h22m42.319566335s) [noise=true , address=77.90.31.15:4700 , piece_id=CGWM3PZIGR2LX63KTTNJALEFI4NY2YVLBAZAXKGK2BE3V37D6GMA, seg_pos={0 0}, upload=274918]: Uploading piece...
Apr 19 14:08:55.501048167 (286h22m42.320284191s) [noise=true , address=145.79.227.3:4154 , piece_id=GCLCPG2QCIT4AMKHMAU4QZHPAONPDNSJAGLLVN4TO2VBZZGJOGIA, seg_pos={0 0}, upload=274918]: Uploading piece...
Apr 19 14:08:55.501263162 (286h22m42.320499186s) [noise=true , address=145.79.227.17:4972 , piece_id=57NAU5R6YCE4AA3PLAKII2U4K5KJCYDS3NH5PZAOUL36W7MSLONQ, seg_pos={0 0}, upload=274918]: Uploading piece...
Apr 19 14:08:57.949098467 (286h22m44.768334501s) [upload=274918]: Next returned nil segment
Apr 19 14:08:57.949115480 (286h22m44.768351554s) [upload=274918]: Waiting for error group managing segments...
Apr 19 14:08:58.345206270 (286h22m45.164442294s) [noise=true , address=145.79.227.3:4154 , piece_id=GCLCPG2QCIT4AMKHMAU4QZHPAONPDNSJAGLLVN4TO2VBZZGJOGIA, seg_pos={0 0}, upload=274918]: Done uploading piece. err: <nil>
Apr 19 14:09:01.082767579 (286h22m47.902003603s) [noise=true , address=45.91.116.11:4152 , piece_id=O46QV2WY5ZRZSU3BJK32J6B2QAYIQH6VPSDB5LVZEHIL6FITZQQQ, seg_pos={0 0}, upload=274918]: Done uploading piece. err: <nil>
Apr 19 14:09:05.145686115 (286h22m51.964922139s) [noise=true , address=77.90.31.16:4738 , piece_id=NOF6IEE7PK5DHZ7FMWTRGL6WJ5I7D5XK7HEZKEUB7SCV6YGW3JQQ, seg_pos={0 0}, upload=274918]: Done uploading piece. err: <nil>
Apr 19 14:09:10.080594511 (286h22m56.899830535s) [noise=true , address=145.79.227.38:4982 , piece_id=ERRKLW4B4WI7YWVD2JJPEDHSU4IEJJM5UHFHTJ3CELIWXNZTJ34Q, seg_pos={0 0}, upload=274918]: Done uploading piece. err: <nil>
Apr 19 14:09:11.441874037 (286h22m58.261110061s) [noise=true , address=145.79.227.15:4879 , piece_id=MWHLQQFFDK2FDKJN7J4MY7FY5O3MB6WMQFXBOLTXMRPRCTSVMQRQ, seg_pos={0 0}, upload=274918]: Done uploading piece. err: <nil>
Apr 19 14:09:24.600269009 (286h23m11.419505053s) [noise=true , address=145.79.227.2:4119 , piece_id=ME7PKNTEYNW36GBEP2LNSMUZMTS7JR67BEUHWEDYTWKB6ANYYHKQ, seg_pos={0 0}, upload=274918]: Done uploading piece. err: ecclient: upload failed (node:121xNAnkm9HZXUszp26UmWTsuhp3igyeDVea6gSCdtrQUQDVHpW, address:145.79.227.2:4119): protocol: expected piece hash; context deadline exceeded; EOF
Apr 19 14:09:24.600415442 (286h23m11.419651496s) [noise=true , address=145.79.227.17:4972 , piece_id=57NAU5R6YCE4AA3PLAKII2U4K5KJCYDS3NH5PZAOUL36W7MSLONQ, seg_pos={0 0}, upload=274918]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: write tcp 188.166.225.170:55998->145.79.227.17:4972: use of closed network connection; write tcp 188.166.225.170:55998->145.79.227.17:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55998->145.79.227.17:4972: use of closed network connection
Apr 19 14:09:24.600668150 (286h23m11.419904204s) [noise=true , address=145.79.227.9:4366 , piece_id=D2KZ4TICJOQGO4FN5ITDMYVH6NH7FAFDKVNDYZTFSGDNGO3AT4FQ, seg_pos={0 0}, upload=274918]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: rpc: dial tcp 145.79.227.9:4366: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 19 14:09:24.600698107 (286h23m11.419934181s) [noise=true , address=145.79.227.36:4518 , piece_id=JC5WKYW543ZUQOAJG33PLM4A6A7WYGWBC6QLYCUKEGSSHYX6JDBA, seg_pos={0 0}, upload=274918]: Done uploading piece. err: ecclient: upload failed (node:17TZ9JLdCivFF46guhSTW6hcoUmTz5uo5wvLkDqC4BGmpm9eLb, address:145.79.227.36:4518): protocol: expected piece hash; context deadline exceeded; EOF
Apr 19 14:09:24.600963510 (286h23m11.420199584s) [noise=true , address=77.90.31.15:4700 , piece_id=CGWM3PZIGR2LX63KTTNJALEFI4NY2YVLBAZAXKGK2BE3V37D6GMA, seg_pos={0 0}, upload=274918]: Done uploading piece. err: ecclient: upload failed (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, address:77.90.31.15:4700): protocol: read tcp 188.166.225.170:48972->77.90.31.15:4700: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 19 14:09:24.601113419 (286h23m11.420349453s) [noise=true , address=145.79.227.34:4907 , piece_id=OVWJEETOWMYMW4OBXTEM2IR44YU2USTJHSKVONV4FMV2LOTFXM5Q, seg_pos={0 0}, upload=274918]: Done uploading piece. err: ecclient: upload failed (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS, address:145.79.227.34:4907): protocol: rpc: dial tcp 145.79.227.34:4907: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 19 14:09:24.601152354 (286h23m11.420388378s) [noise=true , address=45.91.116.7:4012 , piece_id=SDT4HR32F6WKFNFJXHMU4KMUGXUEJ32OYUPOC6SJBTATEXUL5RNQ, seg_pos={0 0}, upload=274918]: Done uploading piece. err: ecclient: upload failed (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, address:45.91.116.7:4012): protocol: read tcp 188.166.225.170:41998->45.91.116.7:4012: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 19 14:09:24.601398209 (286h23m11.420634233s) [noise=true , address=145.79.227.51:4415 , piece_id=RDAXQUREP4SXGB4TCQJUXD6YPRXXBFJIRFEPHUOA7ZYX6A5QOM2A, seg_pos={0 0}, upload=274918]: Done uploading piece. err: ecclient: upload failed (node:129sUhRR5RYtmrrfcEq2KgikZqZqQMRbwgK8NAKLFTTZgXDHW1v, address:145.79.227.51:4415): protocol: read tcp 188.166.225.170:59672->145.79.227.51:4415: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 19 14:09:24.601602834 (286h23m11.420838888s) [noise=true , address=145.79.227.37:4568 , piece_id=XT3EGREWAC2Q6RZXKJAM6ORKDYUSNRIKT5RJOMGEJ4B2N2PEC46A, seg_pos={0 0}, upload=274918]: Done uploading piece. err: ecclient: upload failed (node:1zJigHghNntzHNWim5RnWcMPfeUAbqsrvrfWretYcyEkXbSYzJ, address:145.79.227.37:4568): protocol: read tcp 188.166.225.170:49018->145.79.227.37:4568: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 19 14:09:24.601856011 (286h23m11.421092045s) [noise=true , address=145.79.227.13:4805 , piece_id=GGNMTIQPEMO7RN6KJFGCOUU6XX65CKHFJIZHFC2JVYR2JNJK3Y6Q, seg_pos={0 0}, upload=274918]: Done uploading piece. err: ecclient: upload failed (node:12YEVsEwZwfWsnZFyaSkd9Dn53Tb9EFUyWEVDwTQeMJkcjwrj7x, address:145.79.227.13:4805): protocol: read tcp 188.166.225.170:46898->145.79.227.13:4805: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 19 14:09:24.601963369 (286h23m11.421199393s) [noise=true , address=45.91.116.12:4227 , piece_id=LODLVJMI2EAMJ3EAAOI4Z4RWMOAQHFZL45MSWKVS5MB7FCNJ57AA, seg_pos={0 0}, upload=274918]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: read tcp 188.166.225.170:44102->45.91.116.12:4227: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 19 14:09:24.602008025 (286h23m11.42124405s) [seg_pos={0 0}, upload=274918]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil>
Apr 19 14:09:24.851854994 (286h23m11.671091018s) [upload=274918]: Done uploading segments...
Apr 19 14:09:24.851892576 (286h23m11.6711286s) [upload=274918]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5)
Duration: 5.750587652s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [X...........X...............................................] duration:
3.700843923s trace:
https://jaeger.dev.storj.io/trace/3c23449320be8b26
FAIL (2026-04-19 14:48:42 UTC)
orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded
Duration: 3.700895563s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.........X..................................................] duration:
3.090493252s trace:
https://jaeger.dev.storj.io/trace/2fe350ab9c48d459
FAIL (2026-04-19 14:48:33 UTC)
orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 23)
Duration: 3.090509693s
core
uplink
ap1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.......X....................................................] duration:
5.444505761s trace:
https://jaeger.dev.storj.io/trace/59b3e23b0ad9afe7
FAIL (2026-04-19 14:48:28 UTC)
orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 39)
Duration: 5.444532201s
core
uplink
us1
|
checkers.HTTP{"GET" "https://www.apple.com/robots.txt"} failure (antiblip window 3/60): (http.apple_robots_txt)antiblip:
1 out of 60 is failed (threshold=3) [......X.....................................................] duration:
12.648752ms remote:
"tcp:104.119.104.223:443" target:
https://www.apple.com/robots.txt trace:
https://jaeger.dev.storj.io/trace/505bb4ef80a6b4b1
FAIL (2026-04-19 14:48:23 UTC)
orbiter warning: Get "https://www.apple.com/robots.txt": EOF
Duration: 12.683479ms
core
website
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..........X.................................................] duration:
3.077387697s trace:
https://jaeger.dev.storj.io/trace/2fe350ab9c48d459
FAIL (2026-04-19 14:48:10 UTC)
orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 23)
Duration: 3.077414009s
core
uplink
ap1
|
checkers.HTTP{"GET" "https://www.apple.com/robots.txt"} failure (antiblip window 3/60): (http.apple_robots_txt)antiblip:
1 out of 60 is failed (threshold=3) [.......X....................................................] duration:
20.891389ms remote:
"tcp:104.119.104.223:443" target:
https://www.apple.com/robots.txt trace:
https://jaeger.dev.storj.io/trace/505bb4ef80a6b4b1
FAIL (2026-04-19 14:48:00 UTC)
orbiter warning: Get "https://www.apple.com/robots.txt": EOF
Duration: 20.914805ms
core
website
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [......X.....................................................] duration:
5.788260856s trace:
https://jaeger.dev.storj.io/trace/2e2125521d7abc3e
FAIL (2026-04-19 14:47:57 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 5)
======LOGS======
e...
Apr 19 14:08:55.493690567 (286h22m42.312926591s) [noise=true , address=77.90.31.16:4738 , piece_id=NOF6IEE7PK5DHZ7FMWTRGL6WJ5I7D5XK7HEZKEUB7SCV6YGW3JQQ, seg_pos={0 0}, upload=274918]: Uploading piece...
Apr 19 14:08:55.494428361 (286h22m42.313664385s) [noise=true , address=145.79.227.13:4805 , piece_id=GGNMTIQPEMO7RN6KJFGCOUU6XX65CKHFJIZHFC2JVYR2JNJK3Y6Q, seg_pos={0 0}, upload=274918]: Uploading piece...
Apr 19 14:08:55.494992880 (286h22m42.314228904s) [noise=true , address=45.91.116.12:4227 , piece_id=LODLVJMI2EAMJ3EAAOI4Z4RWMOAQHFZL45MSWKVS5MB7FCNJ57AA, seg_pos={0 0}, upload=274918]: Uploading piece...
Apr 19 14:08:55.495362383 (286h22m42.314598407s) [noise=true , address=145.79.227.51:4415 , piece_id=RDAXQUREP4SXGB4TCQJUXD6YPRXXBFJIRFEPHUOA7ZYX6A5QOM2A, seg_pos={0 0}, upload=274918]: Uploading piece...
Apr 19 14:08:55.495930219 (286h22m42.315166243s) [noise=true , address=45.91.116.11:4152 , piece_id=O46QV2WY5ZRZSU3BJK32J6B2QAYIQH6VPSDB5LVZEHIL6FITZQQQ, seg_pos={0 0}, upload=274918]: Uploading piece...
Apr 19 14:08:55.496278582 (286h22m42.315514606s) [noise=true , address=145.79.227.15:4879 , piece_id=MWHLQQFFDK2FDKJN7J4MY7FY5O3MB6WMQFXBOLTXMRPRCTSVMQRQ, seg_pos={0 0}, upload=274918]: Uploading piece...
Apr 19 14:08:55.497181003 (286h22m42.316417047s) [noise=true , address=145.79.227.9:4366 , piece_id=D2KZ4TICJOQGO4FN5ITDMYVH6NH7FAFDKVNDYZTFSGDNGO3AT4FQ, seg_pos={0 0}, upload=274918]: Uploading piece...
Apr 19 14:08:55.497590223 (286h22m42.316826258s) [noise=true , address=145.79.227.34:4907 , piece_id=OVWJEETOWMYMW4OBXTEM2IR44YU2USTJHSKVONV4FMV2LOTFXM5Q, seg_pos={0 0}, upload=274918]: Uploading piece...
Apr 19 14:08:55.498297648 (286h22m42.317533672s) [noise=true , address=145.79.227.36:4518 , piece_id=JC5WKYW543ZUQOAJG33PLM4A6A7WYGWBC6QLYCUKEGSSHYX6JDBA, seg_pos={0 0}, upload=274918]: Uploading piece...
Apr 19 14:08:55.498945970 (286h22m42.318182014s) [noise=true , address=145.79.227.38:4982 , piece_id=ERRKLW4B4WI7YWVD2JJPEDHSU4IEJJM5UHFHTJ3CELIWXNZTJ34Q, seg_pos={0 0}, upload=274918]: Uploading piece...
Apr 19 14:08:55.499381660 (286h22m42.318617684s) [noise=true , address=145.79.227.2:4119 , piece_id=ME7PKNTEYNW36GBEP2LNSMUZMTS7JR67BEUHWEDYTWKB6ANYYHKQ, seg_pos={0 0}, upload=274918]: Uploading piece...
Apr 19 14:08:55.500079246 (286h22m42.31931527s) [noise=true , address=145.79.227.37:4568 , piece_id=XT3EGREWAC2Q6RZXKJAM6ORKDYUSNRIKT5RJOMGEJ4B2N2PEC46A, seg_pos={0 0}, upload=274918]: Uploading piece...
Apr 19 14:08:55.500330311 (286h22m42.319566335s) [noise=true , address=77.90.31.15:4700 , piece_id=CGWM3PZIGR2LX63KTTNJALEFI4NY2YVLBAZAXKGK2BE3V37D6GMA, seg_pos={0 0}, upload=274918]: Uploading piece...
Apr 19 14:08:55.501048167 (286h22m42.320284191s) [noise=true , address=145.79.227.3:4154 , piece_id=GCLCPG2QCIT4AMKHMAU4QZHPAONPDNSJAGLLVN4TO2VBZZGJOGIA, seg_pos={0 0}, upload=274918]: Uploading piece...
Apr 19 14:08:55.501263162 (286h22m42.320499186s) [noise=true , address=145.79.227.17:4972 , piece_id=57NAU5R6YCE4AA3PLAKII2U4K5KJCYDS3NH5PZAOUL36W7MSLONQ, seg_pos={0 0}, upload=274918]: Uploading piece...
Apr 19 14:08:57.949098467 (286h22m44.768334501s) [upload=274918]: Next returned nil segment
Apr 19 14:08:57.949115480 (286h22m44.768351554s) [upload=274918]: Waiting for error group managing segments...
Apr 19 14:08:58.345206270 (286h22m45.164442294s) [noise=true , address=145.79.227.3:4154 , piece_id=GCLCPG2QCIT4AMKHMAU4QZHPAONPDNSJAGLLVN4TO2VBZZGJOGIA, seg_pos={0 0}, upload=274918]: Done uploading piece. err: <nil>
Apr 19 14:09:01.082767579 (286h22m47.902003603s) [noise=true , address=45.91.116.11:4152 , piece_id=O46QV2WY5ZRZSU3BJK32J6B2QAYIQH6VPSDB5LVZEHIL6FITZQQQ, seg_pos={0 0}, upload=274918]: Done uploading piece. err: <nil>
Apr 19 14:09:05.145686115 (286h22m51.964922139s) [noise=true , address=77.90.31.16:4738 , piece_id=NOF6IEE7PK5DHZ7FMWTRGL6WJ5I7D5XK7HEZKEUB7SCV6YGW3JQQ, seg_pos={0 0}, upload=274918]: Done uploading piece. err: <nil>
Apr 19 14:09:10.080594511 (286h22m56.899830535s) [noise=true , address=145.79.227.38:4982 , piece_id=ERRKLW4B4WI7YWVD2JJPEDHSU4IEJJM5UHFHTJ3CELIWXNZTJ34Q, seg_pos={0 0}, upload=274918]: Done uploading piece. err: <nil>
Apr 19 14:09:11.441874037 (286h22m58.261110061s) [noise=true , address=145.79.227.15:4879 , piece_id=MWHLQQFFDK2FDKJN7J4MY7FY5O3MB6WMQFXBOLTXMRPRCTSVMQRQ, seg_pos={0 0}, upload=274918]: Done uploading piece. err: <nil>
Apr 19 14:09:24.600269009 (286h23m11.419505053s) [noise=true , address=145.79.227.2:4119 , piece_id=ME7PKNTEYNW36GBEP2LNSMUZMTS7JR67BEUHWEDYTWKB6ANYYHKQ, seg_pos={0 0}, upload=274918]: Done uploading piece. err: ecclient: upload failed (node:121xNAnkm9HZXUszp26UmWTsuhp3igyeDVea6gSCdtrQUQDVHpW, address:145.79.227.2:4119): protocol: expected piece hash; context deadline exceeded; EOF
Apr 19 14:09:24.600415442 (286h23m11.419651496s) [noise=true , address=145.79.227.17:4972 , piece_id=57NAU5R6YCE4AA3PLAKII2U4K5KJCYDS3NH5PZAOUL36W7MSLONQ, seg_pos={0 0}, upload=274918]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: write tcp 188.166.225.170:55998->145.79.227.17:4972: use of closed network connection; write tcp 188.166.225.170:55998->145.79.227.17:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55998->145.79.227.17:4972: use of closed network connection
Apr 19 14:09:24.600668150 (286h23m11.419904204s) [noise=true , address=145.79.227.9:4366 , piece_id=D2KZ4TICJOQGO4FN5ITDMYVH6NH7FAFDKVNDYZTFSGDNGO3AT4FQ, seg_pos={0 0}, upload=274918]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: rpc: dial tcp 145.79.227.9:4366: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 19 14:09:24.600698107 (286h23m11.419934181s) [noise=true , address=145.79.227.36:4518 , piece_id=JC5WKYW543ZUQOAJG33PLM4A6A7WYGWBC6QLYCUKEGSSHYX6JDBA, seg_pos={0 0}, upload=274918]: Done uploading piece. err: ecclient: upload failed (node:17TZ9JLdCivFF46guhSTW6hcoUmTz5uo5wvLkDqC4BGmpm9eLb, address:145.79.227.36:4518): protocol: expected piece hash; context deadline exceeded; EOF
Apr 19 14:09:24.600963510 (286h23m11.420199584s) [noise=true , address=77.90.31.15:4700 , piece_id=CGWM3PZIGR2LX63KTTNJALEFI4NY2YVLBAZAXKGK2BE3V37D6GMA, seg_pos={0 0}, upload=274918]: Done uploading piece. err: ecclient: upload failed (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, address:77.90.31.15:4700): protocol: read tcp 188.166.225.170:48972->77.90.31.15:4700: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 19 14:09:24.601113419 (286h23m11.420349453s) [noise=true , address=145.79.227.34:4907 , piece_id=OVWJEETOWMYMW4OBXTEM2IR44YU2USTJHSKVONV4FMV2LOTFXM5Q, seg_pos={0 0}, upload=274918]: Done uploading piece. err: ecclient: upload failed (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS, address:145.79.227.34:4907): protocol: rpc: dial tcp 145.79.227.34:4907: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 19 14:09:24.601152354 (286h23m11.420388378s) [noise=true , address=45.91.116.7:4012 , piece_id=SDT4HR32F6WKFNFJXHMU4KMUGXUEJ32OYUPOC6SJBTATEXUL5RNQ, seg_pos={0 0}, upload=274918]: Done uploading piece. err: ecclient: upload failed (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, address:45.91.116.7:4012): protocol: read tcp 188.166.225.170:41998->45.91.116.7:4012: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 19 14:09:24.601398209 (286h23m11.420634233s) [noise=true , address=145.79.227.51:4415 , piece_id=RDAXQUREP4SXGB4TCQJUXD6YPRXXBFJIRFEPHUOA7ZYX6A5QOM2A, seg_pos={0 0}, upload=274918]: Done uploading piece. err: ecclient: upload failed (node:129sUhRR5RYtmrrfcEq2KgikZqZqQMRbwgK8NAKLFTTZgXDHW1v, address:145.79.227.51:4415): protocol: read tcp 188.166.225.170:59672->145.79.227.51:4415: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 19 14:09:24.601602834 (286h23m11.420838888s) [noise=true , address=145.79.227.37:4568 , piece_id=XT3EGREWAC2Q6RZXKJAM6ORKDYUSNRIKT5RJOMGEJ4B2N2PEC46A, seg_pos={0 0}, upload=274918]: Done uploading piece. err: ecclient: upload failed (node:1zJigHghNntzHNWim5RnWcMPfeUAbqsrvrfWretYcyEkXbSYzJ, address:145.79.227.37:4568): protocol: read tcp 188.166.225.170:49018->145.79.227.37:4568: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 19 14:09:24.601856011 (286h23m11.421092045s) [noise=true , address=145.79.227.13:4805 , piece_id=GGNMTIQPEMO7RN6KJFGCOUU6XX65CKHFJIZHFC2JVYR2JNJK3Y6Q, seg_pos={0 0}, upload=274918]: Done uploading piece. err: ecclient: upload failed (node:12YEVsEwZwfWsnZFyaSkd9Dn53Tb9EFUyWEVDwTQeMJkcjwrj7x, address:145.79.227.13:4805): protocol: read tcp 188.166.225.170:46898->145.79.227.13:4805: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 19 14:09:24.601963369 (286h23m11.421199393s) [noise=true , address=45.91.116.12:4227 , piece_id=LODLVJMI2EAMJ3EAAOI4Z4RWMOAQHFZL45MSWKVS5MB7FCNJ57AA, seg_pos={0 0}, upload=274918]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: read tcp 188.166.225.170:44102->45.91.116.12:4227: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 19 14:09:24.602008025 (286h23m11.42124405s) [seg_pos={0 0}, upload=274918]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil>
Apr 19 14:09:24.851854994 (286h23m11.671091018s) [upload=274918]: Done uploading segments...
Apr 19 14:09:24.851892576 (286h23m11.6711286s) [upload=274918]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5)
Duration: 5.788276738s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........X................................................] duration:
2.933980331s trace:
https://jaeger.dev.storj.io/trace/2fe350ab9c48d459
FAIL (2026-04-19 14:47:53 UTC)
orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 23)
Duration: 2.934006012s
core
uplink
ap1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [.X...........X..............................................] duration:
3.531314591s trace:
https://jaeger.dev.storj.io/trace/3c23449320be8b26
FAIL (2026-04-19 14:47:49 UTC)
orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded
Duration: 3.531326655s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.......X....................................................] duration:
5.777287415s trace:
https://jaeger.dev.storj.io/trace/2e2125521d7abc3e
FAIL (2026-04-19 14:47:38 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 5)
======LOGS======
e...
Apr 19 14:08:55.493690567 (286h22m42.312926591s) [noise=true , address=77.90.31.16:4738 , piece_id=NOF6IEE7PK5DHZ7FMWTRGL6WJ5I7D5XK7HEZKEUB7SCV6YGW3JQQ, seg_pos={0 0}, upload=274918]: Uploading piece...
Apr 19 14:08:55.494428361 (286h22m42.313664385s) [noise=true , address=145.79.227.13:4805 , piece_id=GGNMTIQPEMO7RN6KJFGCOUU6XX65CKHFJIZHFC2JVYR2JNJK3Y6Q, seg_pos={0 0}, upload=274918]: Uploading piece...
Apr 19 14:08:55.494992880 (286h22m42.314228904s) [noise=true , address=45.91.116.12:4227 , piece_id=LODLVJMI2EAMJ3EAAOI4Z4RWMOAQHFZL45MSWKVS5MB7FCNJ57AA, seg_pos={0 0}, upload=274918]: Uploading piece...
Apr 19 14:08:55.495362383 (286h22m42.314598407s) [noise=true , address=145.79.227.51:4415 , piece_id=RDAXQUREP4SXGB4TCQJUXD6YPRXXBFJIRFEPHUOA7ZYX6A5QOM2A, seg_pos={0 0}, upload=274918]: Uploading piece...
Apr 19 14:08:55.495930219 (286h22m42.315166243s) [noise=true , address=45.91.116.11:4152 , piece_id=O46QV2WY5ZRZSU3BJK32J6B2QAYIQH6VPSDB5LVZEHIL6FITZQQQ, seg_pos={0 0}, upload=274918]: Uploading piece...
Apr 19 14:08:55.496278582 (286h22m42.315514606s) [noise=true , address=145.79.227.15:4879 , piece_id=MWHLQQFFDK2FDKJN7J4MY7FY5O3MB6WMQFXBOLTXMRPRCTSVMQRQ, seg_pos={0 0}, upload=274918]: Uploading piece...
Apr 19 14:08:55.497181003 (286h22m42.316417047s) [noise=true , address=145.79.227.9:4366 , piece_id=D2KZ4TICJOQGO4FN5ITDMYVH6NH7FAFDKVNDYZTFSGDNGO3AT4FQ, seg_pos={0 0}, upload=274918]: Uploading piece...
Apr 19 14:08:55.497590223 (286h22m42.316826258s) [noise=true , address=145.79.227.34:4907 , piece_id=OVWJEETOWMYMW4OBXTEM2IR44YU2USTJHSKVONV4FMV2LOTFXM5Q, seg_pos={0 0}, upload=274918]: Uploading piece...
Apr 19 14:08:55.498297648 (286h22m42.317533672s) [noise=true , address=145.79.227.36:4518 , piece_id=JC5WKYW543ZUQOAJG33PLM4A6A7WYGWBC6QLYCUKEGSSHYX6JDBA, seg_pos={0 0}, upload=274918]: Uploading piece...
Apr 19 14:08:55.498945970 (286h22m42.318182014s) [noise=true , address=145.79.227.38:4982 , piece_id=ERRKLW4B4WI7YWVD2JJPEDHSU4IEJJM5UHFHTJ3CELIWXNZTJ34Q, seg_pos={0 0}, upload=274918]: Uploading piece...
Apr 19 14:08:55.499381660 (286h22m42.318617684s) [noise=true , address=145.79.227.2:4119 , piece_id=ME7PKNTEYNW36GBEP2LNSMUZMTS7JR67BEUHWEDYTWKB6ANYYHKQ, seg_pos={0 0}, upload=274918]: Uploading piece...
Apr 19 14:08:55.500079246 (286h22m42.31931527s) [noise=true , address=145.79.227.37:4568 , piece_id=XT3EGREWAC2Q6RZXKJAM6ORKDYUSNRIKT5RJOMGEJ4B2N2PEC46A, seg_pos={0 0}, upload=274918]: Uploading piece...
Apr 19 14:08:55.500330311 (286h22m42.319566335s) [noise=true , address=77.90.31.15:4700 , piece_id=CGWM3PZIGR2LX63KTTNJALEFI4NY2YVLBAZAXKGK2BE3V37D6GMA, seg_pos={0 0}, upload=274918]: Uploading piece...
Apr 19 14:08:55.501048167 (286h22m42.320284191s) [noise=true , address=145.79.227.3:4154 , piece_id=GCLCPG2QCIT4AMKHMAU4QZHPAONPDNSJAGLLVN4TO2VBZZGJOGIA, seg_pos={0 0}, upload=274918]: Uploading piece...
Apr 19 14:08:55.501263162 (286h22m42.320499186s) [noise=true , address=145.79.227.17:4972 , piece_id=57NAU5R6YCE4AA3PLAKII2U4K5KJCYDS3NH5PZAOUL36W7MSLONQ, seg_pos={0 0}, upload=274918]: Uploading piece...
Apr 19 14:08:57.949098467 (286h22m44.768334501s) [upload=274918]: Next returned nil segment
Apr 19 14:08:57.949115480 (286h22m44.768351554s) [upload=274918]: Waiting for error group managing segments...
Apr 19 14:08:58.345206270 (286h22m45.164442294s) [noise=true , address=145.79.227.3:4154 , piece_id=GCLCPG2QCIT4AMKHMAU4QZHPAONPDNSJAGLLVN4TO2VBZZGJOGIA, seg_pos={0 0}, upload=274918]: Done uploading piece. err: <nil>
Apr 19 14:09:01.082767579 (286h22m47.902003603s) [noise=true , address=45.91.116.11:4152 , piece_id=O46QV2WY5ZRZSU3BJK32J6B2QAYIQH6VPSDB5LVZEHIL6FITZQQQ, seg_pos={0 0}, upload=274918]: Done uploading piece. err: <nil>
Apr 19 14:09:05.145686115 (286h22m51.964922139s) [noise=true , address=77.90.31.16:4738 , piece_id=NOF6IEE7PK5DHZ7FMWTRGL6WJ5I7D5XK7HEZKEUB7SCV6YGW3JQQ, seg_pos={0 0}, upload=274918]: Done uploading piece. err: <nil>
Apr 19 14:09:10.080594511 (286h22m56.899830535s) [noise=true , address=145.79.227.38:4982 , piece_id=ERRKLW4B4WI7YWVD2JJPEDHSU4IEJJM5UHFHTJ3CELIWXNZTJ34Q, seg_pos={0 0}, upload=274918]: Done uploading piece. err: <nil>
Apr 19 14:09:11.441874037 (286h22m58.261110061s) [noise=true , address=145.79.227.15:4879 , piece_id=MWHLQQFFDK2FDKJN7J4MY7FY5O3MB6WMQFXBOLTXMRPRCTSVMQRQ, seg_pos={0 0}, upload=274918]: Done uploading piece. err: <nil>
Apr 19 14:09:24.600269009 (286h23m11.419505053s) [noise=true , address=145.79.227.2:4119 , piece_id=ME7PKNTEYNW36GBEP2LNSMUZMTS7JR67BEUHWEDYTWKB6ANYYHKQ, seg_pos={0 0}, upload=274918]: Done uploading piece. err: ecclient: upload failed (node:121xNAnkm9HZXUszp26UmWTsuhp3igyeDVea6gSCdtrQUQDVHpW, address:145.79.227.2:4119): protocol: expected piece hash; context deadline exceeded; EOF
Apr 19 14:09:24.600415442 (286h23m11.419651496s) [noise=true , address=145.79.227.17:4972 , piece_id=57NAU5R6YCE4AA3PLAKII2U4K5KJCYDS3NH5PZAOUL36W7MSLONQ, seg_pos={0 0}, upload=274918]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: write tcp 188.166.225.170:55998->145.79.227.17:4972: use of closed network connection; write tcp 188.166.225.170:55998->145.79.227.17:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55998->145.79.227.17:4972: use of closed network connection
Apr 19 14:09:24.600668150 (286h23m11.419904204s) [noise=true , address=145.79.227.9:4366 , piece_id=D2KZ4TICJOQGO4FN5ITDMYVH6NH7FAFDKVNDYZTFSGDNGO3AT4FQ, seg_pos={0 0}, upload=274918]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: rpc: dial tcp 145.79.227.9:4366: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 19 14:09:24.600698107 (286h23m11.419934181s) [noise=true , address=145.79.227.36:4518 , piece_id=JC5WKYW543ZUQOAJG33PLM4A6A7WYGWBC6QLYCUKEGSSHYX6JDBA, seg_pos={0 0}, upload=274918]: Done uploading piece. err: ecclient: upload failed (node:17TZ9JLdCivFF46guhSTW6hcoUmTz5uo5wvLkDqC4BGmpm9eLb, address:145.79.227.36:4518): protocol: expected piece hash; context deadline exceeded; EOF
Apr 19 14:09:24.600963510 (286h23m11.420199584s) [noise=true , address=77.90.31.15:4700 , piece_id=CGWM3PZIGR2LX63KTTNJALEFI4NY2YVLBAZAXKGK2BE3V37D6GMA, seg_pos={0 0}, upload=274918]: Done uploading piece. err: ecclient: upload failed (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, address:77.90.31.15:4700): protocol: read tcp 188.166.225.170:48972->77.90.31.15:4700: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 19 14:09:24.601113419 (286h23m11.420349453s) [noise=true , address=145.79.227.34:4907 , piece_id=OVWJEETOWMYMW4OBXTEM2IR44YU2USTJHSKVONV4FMV2LOTFXM5Q, seg_pos={0 0}, upload=274918]: Done uploading piece. err: ecclient: upload failed (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS, address:145.79.227.34:4907): protocol: rpc: dial tcp 145.79.227.34:4907: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 19 14:09:24.601152354 (286h23m11.420388378s) [noise=true , address=45.91.116.7:4012 , piece_id=SDT4HR32F6WKFNFJXHMU4KMUGXUEJ32OYUPOC6SJBTATEXUL5RNQ, seg_pos={0 0}, upload=274918]: Done uploading piece. err: ecclient: upload failed (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, address:45.91.116.7:4012): protocol: read tcp 188.166.225.170:41998->45.91.116.7:4012: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 19 14:09:24.601398209 (286h23m11.420634233s) [noise=true , address=145.79.227.51:4415 , piece_id=RDAXQUREP4SXGB4TCQJUXD6YPRXXBFJIRFEPHUOA7ZYX6A5QOM2A, seg_pos={0 0}, upload=274918]: Done uploading piece. err: ecclient: upload failed (node:129sUhRR5RYtmrrfcEq2KgikZqZqQMRbwgK8NAKLFTTZgXDHW1v, address:145.79.227.51:4415): protocol: read tcp 188.166.225.170:59672->145.79.227.51:4415: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 19 14:09:24.601602834 (286h23m11.420838888s) [noise=true , address=145.79.227.37:4568 , piece_id=XT3EGREWAC2Q6RZXKJAM6ORKDYUSNRIKT5RJOMGEJ4B2N2PEC46A, seg_pos={0 0}, upload=274918]: Done uploading piece. err: ecclient: upload failed (node:1zJigHghNntzHNWim5RnWcMPfeUAbqsrvrfWretYcyEkXbSYzJ, address:145.79.227.37:4568): protocol: read tcp 188.166.225.170:49018->145.79.227.37:4568: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 19 14:09:24.601856011 (286h23m11.421092045s) [noise=true , address=145.79.227.13:4805 , piece_id=GGNMTIQPEMO7RN6KJFGCOUU6XX65CKHFJIZHFC2JVYR2JNJK3Y6Q, seg_pos={0 0}, upload=274918]: Done uploading piece. err: ecclient: upload failed (node:12YEVsEwZwfWsnZFyaSkd9Dn53Tb9EFUyWEVDwTQeMJkcjwrj7x, address:145.79.227.13:4805): protocol: read tcp 188.166.225.170:46898->145.79.227.13:4805: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 19 14:09:24.601963369 (286h23m11.421199393s) [noise=true , address=45.91.116.12:4227 , piece_id=LODLVJMI2EAMJ3EAAOI4Z4RWMOAQHFZL45MSWKVS5MB7FCNJ57AA, seg_pos={0 0}, upload=274918]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: read tcp 188.166.225.170:44102->45.91.116.12:4227: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 19 14:09:24.602008025 (286h23m11.42124405s) [seg_pos={0 0}, upload=274918]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil>
Apr 19 14:09:24.851854994 (286h23m11.671091018s) [upload=274918]: Done uploading segments...
Apr 19 14:09:24.851892576 (286h23m11.6711286s) [upload=274918]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5)
Duration: 5.777326861s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
1 out of 60 is failed (threshold=3) [........X...................................................] duration:
5.31840442s trace:
https://jaeger.dev.storj.io/trace/59b3e23b0ad9afe7
FAIL (2026-04-19 14:47:37 UTC)
orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 39)
Duration: 5.318433335s
core
uplink
us1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [..X...........X.............................................] duration:
3.533254093s trace:
https://jaeger.dev.storj.io/trace/3c23449320be8b26
FAIL (2026-04-19 14:47:31 UTC)
orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded
Duration: 3.533281225s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.........X..................................................] duration:
5.21231628s trace:
https://jaeger.dev.storj.io/trace/59b3e23b0ad9afe7
FAIL (2026-04-19 14:47:11 UTC)
orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 39)
Duration: 5.212340547s
core
uplink
us1
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.remote)antiblip:
1 out of 60 is failed (threshold=3) [............X...............................................] duration:
3.190445643s trace:
https://jaeger.dev.storj.io/trace/2fe350ab9c48d459
FAIL (2026-04-19 14:47:10 UTC)
orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 23)
Duration: 3.190475732s
core
uplink
ap1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [........X...................................................] duration:
5.776339991s trace:
https://jaeger.dev.storj.io/trace/2e2125521d7abc3e
FAIL (2026-04-19 14:46:52 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 5)
======LOGS======
e...
Apr 19 14:08:55.493690567 (286h22m42.312926591s) [noise=true , address=77.90.31.16:4738 , piece_id=NOF6IEE7PK5DHZ7FMWTRGL6WJ5I7D5XK7HEZKEUB7SCV6YGW3JQQ, seg_pos={0 0}, upload=274918]: Uploading piece...
Apr 19 14:08:55.494428361 (286h22m42.313664385s) [noise=true , address=145.79.227.13:4805 , piece_id=GGNMTIQPEMO7RN6KJFGCOUU6XX65CKHFJIZHFC2JVYR2JNJK3Y6Q, seg_pos={0 0}, upload=274918]: Uploading piece...
Apr 19 14:08:55.494992880 (286h22m42.314228904s) [noise=true , address=45.91.116.12:4227 , piece_id=LODLVJMI2EAMJ3EAAOI4Z4RWMOAQHFZL45MSWKVS5MB7FCNJ57AA, seg_pos={0 0}, upload=274918]: Uploading piece...
Apr 19 14:08:55.495362383 (286h22m42.314598407s) [noise=true , address=145.79.227.51:4415 , piece_id=RDAXQUREP4SXGB4TCQJUXD6YPRXXBFJIRFEPHUOA7ZYX6A5QOM2A, seg_pos={0 0}, upload=274918]: Uploading piece...
Apr 19 14:08:55.495930219 (286h22m42.315166243s) [noise=true , address=45.91.116.11:4152 , piece_id=O46QV2WY5ZRZSU3BJK32J6B2QAYIQH6VPSDB5LVZEHIL6FITZQQQ, seg_pos={0 0}, upload=274918]: Uploading piece...
Apr 19 14:08:55.496278582 (286h22m42.315514606s) [noise=true , address=145.79.227.15:4879 , piece_id=MWHLQQFFDK2FDKJN7J4MY7FY5O3MB6WMQFXBOLTXMRPRCTSVMQRQ, seg_pos={0 0}, upload=274918]: Uploading piece...
Apr 19 14:08:55.497181003 (286h22m42.316417047s) [noise=true , address=145.79.227.9:4366 , piece_id=D2KZ4TICJOQGO4FN5ITDMYVH6NH7FAFDKVNDYZTFSGDNGO3AT4FQ, seg_pos={0 0}, upload=274918]: Uploading piece...
Apr 19 14:08:55.497590223 (286h22m42.316826258s) [noise=true , address=145.79.227.34:4907 , piece_id=OVWJEETOWMYMW4OBXTEM2IR44YU2USTJHSKVONV4FMV2LOTFXM5Q, seg_pos={0 0}, upload=274918]: Uploading piece...
Apr 19 14:08:55.498297648 (286h22m42.317533672s) [noise=true , address=145.79.227.36:4518 , piece_id=JC5WKYW543ZUQOAJG33PLM4A6A7WYGWBC6QLYCUKEGSSHYX6JDBA, seg_pos={0 0}, upload=274918]: Uploading piece...
Apr 19 14:08:55.498945970 (286h22m42.318182014s) [noise=true , address=145.79.227.38:4982 , piece_id=ERRKLW4B4WI7YWVD2JJPEDHSU4IEJJM5UHFHTJ3CELIWXNZTJ34Q, seg_pos={0 0}, upload=274918]: Uploading piece...
Apr 19 14:08:55.499381660 (286h22m42.318617684s) [noise=true , address=145.79.227.2:4119 , piece_id=ME7PKNTEYNW36GBEP2LNSMUZMTS7JR67BEUHWEDYTWKB6ANYYHKQ, seg_pos={0 0}, upload=274918]: Uploading piece...
Apr 19 14:08:55.500079246 (286h22m42.31931527s) [noise=true , address=145.79.227.37:4568 , piece_id=XT3EGREWAC2Q6RZXKJAM6ORKDYUSNRIKT5RJOMGEJ4B2N2PEC46A, seg_pos={0 0}, upload=274918]: Uploading piece...
Apr 19 14:08:55.500330311 (286h22m42.319566335s) [noise=true , address=77.90.31.15:4700 , piece_id=CGWM3PZIGR2LX63KTTNJALEFI4NY2YVLBAZAXKGK2BE3V37D6GMA, seg_pos={0 0}, upload=274918]: Uploading piece...
Apr 19 14:08:55.501048167 (286h22m42.320284191s) [noise=true , address=145.79.227.3:4154 , piece_id=GCLCPG2QCIT4AMKHMAU4QZHPAONPDNSJAGLLVN4TO2VBZZGJOGIA, seg_pos={0 0}, upload=274918]: Uploading piece...
Apr 19 14:08:55.501263162 (286h22m42.320499186s) [noise=true , address=145.79.227.17:4972 , piece_id=57NAU5R6YCE4AA3PLAKII2U4K5KJCYDS3NH5PZAOUL36W7MSLONQ, seg_pos={0 0}, upload=274918]: Uploading piece...
Apr 19 14:08:57.949098467 (286h22m44.768334501s) [upload=274918]: Next returned nil segment
Apr 19 14:08:57.949115480 (286h22m44.768351554s) [upload=274918]: Waiting for error group managing segments...
Apr 19 14:08:58.345206270 (286h22m45.164442294s) [noise=true , address=145.79.227.3:4154 , piece_id=GCLCPG2QCIT4AMKHMAU4QZHPAONPDNSJAGLLVN4TO2VBZZGJOGIA, seg_pos={0 0}, upload=274918]: Done uploading piece. err: <nil>
Apr 19 14:09:01.082767579 (286h22m47.902003603s) [noise=true , address=45.91.116.11:4152 , piece_id=O46QV2WY5ZRZSU3BJK32J6B2QAYIQH6VPSDB5LVZEHIL6FITZQQQ, seg_pos={0 0}, upload=274918]: Done uploading piece. err: <nil>
Apr 19 14:09:05.145686115 (286h22m51.964922139s) [noise=true , address=77.90.31.16:4738 , piece_id=NOF6IEE7PK5DHZ7FMWTRGL6WJ5I7D5XK7HEZKEUB7SCV6YGW3JQQ, seg_pos={0 0}, upload=274918]: Done uploading piece. err: <nil>
Apr 19 14:09:10.080594511 (286h22m56.899830535s) [noise=true , address=145.79.227.38:4982 , piece_id=ERRKLW4B4WI7YWVD2JJPEDHSU4IEJJM5UHFHTJ3CELIWXNZTJ34Q, seg_pos={0 0}, upload=274918]: Done uploading piece. err: <nil>
Apr 19 14:09:11.441874037 (286h22m58.261110061s) [noise=true , address=145.79.227.15:4879 , piece_id=MWHLQQFFDK2FDKJN7J4MY7FY5O3MB6WMQFXBOLTXMRPRCTSVMQRQ, seg_pos={0 0}, upload=274918]: Done uploading piece. err: <nil>
Apr 19 14:09:24.600269009 (286h23m11.419505053s) [noise=true , address=145.79.227.2:4119 , piece_id=ME7PKNTEYNW36GBEP2LNSMUZMTS7JR67BEUHWEDYTWKB6ANYYHKQ, seg_pos={0 0}, upload=274918]: Done uploading piece. err: ecclient: upload failed (node:121xNAnkm9HZXUszp26UmWTsuhp3igyeDVea6gSCdtrQUQDVHpW, address:145.79.227.2:4119): protocol: expected piece hash; context deadline exceeded; EOF
Apr 19 14:09:24.600415442 (286h23m11.419651496s) [noise=true , address=145.79.227.17:4972 , piece_id=57NAU5R6YCE4AA3PLAKII2U4K5KJCYDS3NH5PZAOUL36W7MSLONQ, seg_pos={0 0}, upload=274918]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: write tcp 188.166.225.170:55998->145.79.227.17:4972: use of closed network connection; write tcp 188.166.225.170:55998->145.79.227.17:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55998->145.79.227.17:4972: use of closed network connection
Apr 19 14:09:24.600668150 (286h23m11.419904204s) [noise=true , address=145.79.227.9:4366 , piece_id=D2KZ4TICJOQGO4FN5ITDMYVH6NH7FAFDKVNDYZTFSGDNGO3AT4FQ, seg_pos={0 0}, upload=274918]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: rpc: dial tcp 145.79.227.9:4366: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 19 14:09:24.600698107 (286h23m11.419934181s) [noise=true , address=145.79.227.36:4518 , piece_id=JC5WKYW543ZUQOAJG33PLM4A6A7WYGWBC6QLYCUKEGSSHYX6JDBA, seg_pos={0 0}, upload=274918]: Done uploading piece. err: ecclient: upload failed (node:17TZ9JLdCivFF46guhSTW6hcoUmTz5uo5wvLkDqC4BGmpm9eLb, address:145.79.227.36:4518): protocol: expected piece hash; context deadline exceeded; EOF
Apr 19 14:09:24.600963510 (286h23m11.420199584s) [noise=true , address=77.90.31.15:4700 , piece_id=CGWM3PZIGR2LX63KTTNJALEFI4NY2YVLBAZAXKGK2BE3V37D6GMA, seg_pos={0 0}, upload=274918]: Done uploading piece. err: ecclient: upload failed (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, address:77.90.31.15:4700): protocol: read tcp 188.166.225.170:48972->77.90.31.15:4700: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 19 14:09:24.601113419 (286h23m11.420349453s) [noise=true , address=145.79.227.34:4907 , piece_id=OVWJEETOWMYMW4OBXTEM2IR44YU2USTJHSKVONV4FMV2LOTFXM5Q, seg_pos={0 0}, upload=274918]: Done uploading piece. err: ecclient: upload failed (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS, address:145.79.227.34:4907): protocol: rpc: dial tcp 145.79.227.34:4907: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 19 14:09:24.601152354 (286h23m11.420388378s) [noise=true , address=45.91.116.7:4012 , piece_id=SDT4HR32F6WKFNFJXHMU4KMUGXUEJ32OYUPOC6SJBTATEXUL5RNQ, seg_pos={0 0}, upload=274918]: Done uploading piece. err: ecclient: upload failed (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, address:45.91.116.7:4012): protocol: read tcp 188.166.225.170:41998->45.91.116.7:4012: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 19 14:09:24.601398209 (286h23m11.420634233s) [noise=true , address=145.79.227.51:4415 , piece_id=RDAXQUREP4SXGB4TCQJUXD6YPRXXBFJIRFEPHUOA7ZYX6A5QOM2A, seg_pos={0 0}, upload=274918]: Done uploading piece. err: ecclient: upload failed (node:129sUhRR5RYtmrrfcEq2KgikZqZqQMRbwgK8NAKLFTTZgXDHW1v, address:145.79.227.51:4415): protocol: read tcp 188.166.225.170:59672->145.79.227.51:4415: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 19 14:09:24.601602834 (286h23m11.420838888s) [noise=true , address=145.79.227.37:4568 , piece_id=XT3EGREWAC2Q6RZXKJAM6ORKDYUSNRIKT5RJOMGEJ4B2N2PEC46A, seg_pos={0 0}, upload=274918]: Done uploading piece. err: ecclient: upload failed (node:1zJigHghNntzHNWim5RnWcMPfeUAbqsrvrfWretYcyEkXbSYzJ, address:145.79.227.37:4568): protocol: read tcp 188.166.225.170:49018->145.79.227.37:4568: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 19 14:09:24.601856011 (286h23m11.421092045s) [noise=true , address=145.79.227.13:4805 , piece_id=GGNMTIQPEMO7RN6KJFGCOUU6XX65CKHFJIZHFC2JVYR2JNJK3Y6Q, seg_pos={0 0}, upload=274918]: Done uploading piece. err: ecclient: upload failed (node:12YEVsEwZwfWsnZFyaSkd9Dn53Tb9EFUyWEVDwTQeMJkcjwrj7x, address:145.79.227.13:4805): protocol: read tcp 188.166.225.170:46898->145.79.227.13:4805: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 19 14:09:24.601963369 (286h23m11.421199393s) [noise=true , address=45.91.116.12:4227 , piece_id=LODLVJMI2EAMJ3EAAOI4Z4RWMOAQHFZL45MSWKVS5MB7FCNJ57AA, seg_pos={0 0}, upload=274918]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: read tcp 188.166.225.170:44102->45.91.116.12:4227: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 19 14:09:24.602008025 (286h23m11.42124405s) [seg_pos={0 0}, upload=274918]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil>
Apr 19 14:09:24.851854994 (286h23m11.671091018s) [upload=274918]: Done uploading segments...
Apr 19 14:09:24.851892576 (286h23m11.6711286s) [upload=274918]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5)
Duration: 5.776369709s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..........X.................................................] duration:
5.511449639s trace:
https://jaeger.dev.storj.io/trace/59b3e23b0ad9afe7
FAIL (2026-04-19 14:46:52 UTC)
orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 39)
Duration: 5.511472114s
core
uplink
us1
|
checkers.HTTP{"GET" "https://www.apple.com/robots.txt"} failure (antiblip window 3/60): (http.apple_robots_txt)antiblip:
1 out of 60 is failed (threshold=3) [........X...................................................] duration:
268.514797ms remote:
"tcp:104.119.104.223:443" target:
https://www.apple.com/robots.txt trace:
https://jaeger.dev.storj.io/trace/505bb4ef80a6b4b1
FAIL (2026-04-19 14:46:49 UTC)
orbiter warning: Get "https://www.apple.com/robots.txt": EOF
Duration: 268.538262ms
core
website
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.............X..............................................] duration:
3.073103661s trace:
https://jaeger.dev.storj.io/trace/2fe350ab9c48d459
FAIL (2026-04-19 14:46:40 UTC)
orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 23)
Duration: 3.073129329s
core
uplink
ap1
|